Page 1 of 1

Scraping

Posted: Thu Apr 13, 2017 10:16 am
by rgb
I think that scraping could be better.
I hope that retrolux could become like plex for videogames.

This is my wish:
I'd like to organize my game and have all the possible information preferably in my language: Italian/english.
I prefer to have video preview of the game so why not import a video from emumovies or download user collection videos.
Better if all information stay outside the the roms directory; why not in my personal cloud ?

Re: Scraping

Posted: Thu Apr 13, 2017 10:29 am
by steviebwoy
Yeah, I agree. I think scraping could be a lot better. I don't mind where this information is held though, I'd be happy for it to be downloaded to my Nvidia Shield rather than the Cloud.

Re: Scraping

Posted: Thu Apr 13, 2017 2:21 pm
by fcatrin
The main problem of scraping in general is:

a) Dependency on other's services
b) Initialization time

Currently we get the data from our own servers but we are far from covering all the games available. We do it on demand so users can start playing as soon as possible. If we get the same data from third parties we should cover the costs and we don't think that is possible now. I've reviewed some experiences about scraping from the Nostalgia and LaunchBox developers and it is not good. Don't take me wrong, it is good for the user, but a big problem for developers.

About video previews I still don't know how good could it work on Android, please check this for background info:

viewtopic.php?f=2&t=3#p16

I've been thinking about improving the way we collect the data about the games. Until now there was one person (Javier) creating all the data, covers and screenshots, but it will be a lot better if I open that part to the community so anybody can submit data for their favorite games and systems.

About cloud storage for metadata, the current implementation of public clouds (gdrive, dropbox) doesn't work as good as I thought, so we are evaluating new options on it.

Re: Scraping

Posted: Thu Apr 13, 2017 9:34 pm
by steviebwoy
I'd be fully supportive of this becoming a community effort. Many hands make light work :)

Re: Scraping

Posted: Sun Aug 27, 2017 4:39 pm
by XBEOX
Is this being updated? I am getting no 3DO scrapes and lots of gaps in general

Great work overall of course

Re: Scraping

Posted: Sun Aug 27, 2017 8:37 pm
by Shane_R_Monroe
I put my hand up to help as well.