Welcome to the Invelos forums. Please read the forum rules before posting.

Read access to our public forums is open to everyone. To post messages, a free registration is required.

If you have an Invelos account, sign in to post.

    Invelos Forums->General: General Discussion Page: 1 2 3 ...5  Previous   Next
Creating my own movie database with titles/cast/crew and everything
Author Message
DVD Profiler Unlimited Registrantmovie_madness
Registered: August 7, 2007
Posts: 102
Posted:
PM this userView this user's DVD collectionDirect link to this postReply with quote
DVD Profiler is nice and all but if you're like me, you also have laserdiscs, VHS, Video CDs, DVR recordings, and online movie downloads, and DVD Profiler can't include them.  I have a considerable amount of movies not on either DVD or Blu-ray.  And I have tons of TV shows that will never ever be on any home video format at all (Oscar shows, rare TCM movies, and the likes).  In my case, DVD Profiler just seems woefully inadequate to catalog my collection.  So I decided to take matters into my own hands.  It had to involve IMDb, since it is the only possible place for me get info on any title conceivable.

If you have programming skills then you must have thought of doing something like this, or may have already done it.  First, I installed the Python programming language.  Then I installed IMDbPY, a set of Python scripts that can retrieve data from IMDb.  I modified the scripts to suit my particular needs.  I created a text file of all my movie titles in my collection.  I imported the file to a Python script, and got a list of IMDb movie IDs (e.g. 0157059) in return.  I used the movie IDs and ran another script, and got a list of cast and crew of all those movies.  I transferred all the IMDb data I got to my choice of database program (in my case, Microsoft Access).  Lastly, I wrote a few vbscript web pages that connects to the Access database.  The webpages let me search for names and return the info I want.

I had thought this project would take me ages, but it didn't!  I have 5000+ movies and TV shows in my collection and it only took me a week and half to get the necessary IMDb data for about 4000 shows into my database.  And the process was not entirely automatic either - I had to look up IMDb movie ID one by one.  But IMDbPY performed pretty speedily.  It took only 1-2 seconds to look up an ID.  It took just less than an hour to retrieve the complete cast and crew of a THOUSAND movies.

Of course, I had looked for third-party software that can import IMDb data but none worked as I had wanted, mostly because they retrieved incomplete data.  With IMDbPY, I was able to retrieve COMPLETE cast and crew, complete everything, just like you see them on the website.

Has anyone here actually done this??  Now I'm able to say that I have.
DVD Profiler Unlimited RegistrantStar ContributorNexus the Sixth
Contributor since 2002
Registered: March 13, 2007
Reputation: High Rating
Sweden Posts: 3,197
Posted:
PM this userView this user's DVD collectionDirect link to this postReply with quote
So you have succeeded in creating another IMDb clone... You're definitely not the first nor the last person to do so, but personally I fail to see what the point is.

DVD Profiler is unique in that it tries to capture what is actually on the disc/film credits. Of course it requires more work than simply scraping data that is already available on the net. But if one doesn't care about the quality or source I guess the latter is acceptable for some people. I use IMDb myself and each tool has its use but I wouldn't like to mix them.
First registered: February 15, 2002
 Last edited: by Nexus the Sixth
DVD Profiler Unlimited RegistrantStar Contributorscotthm
Registered: March 20, 2007
Reputation: Great Rating
United States Posts: 2,851
Posted:
PM this userView this user's DVD collectionDirect link to this postReply with quote
Quoting movie_madness:
Quote:
DVD Profiler is nice and all but if you're like me, you also have laserdiscs, VHS, Video CDs, DVR recordings, and online movie downloads, and DVD Profiler can't include them.

Really?  It includes mine.

---------------
DVD Profiler Unlimited RegistrantStar ContributorWinston Smith
Don't be discommodious
Registered: March 13, 2007
United States Posts: 21,610
Posted:
PM this userEmail this userView this user's DVD collectionDirect link to this postReply with quote
As Kino said you are not the first or the last to clone IMDb. The single most inaccurate database in the World, for the reasons Kino alluded to. They have or had no standards for data entry.

Be warned should you try to market your db amazon/IMDb will say pay  up or else. Their minimum licensing fee for use of their data, inaccurate as it is, is $15,000/yr.

Good luck, Jim. If you are caught or killed the secretary will disavow any  knowledge of your actions.
ASSUME NOTHING!!!!!!
CBE, MBE, MoA and proud of it.
Outta here

Billy Video
DVD Profiler Unlimited RegistrantStar ContributorWinston Smith
Don't be discommodious
Registered: March 13, 2007
United States Posts: 21,610
Posted:
PM this userEmail this userView this user's DVD collectionDirect link to this postReply with quote
Quoting scotthm:
Quote:
Quoting movie_madness:
Quote:
DVD Profiler is nice and all but if you're like me, you also have laserdiscs, VHS, Video CDs, DVR recordings, and online movie downloads, and DVD Profiler can't include them.

Really?  It includes mine.

---------------

Locally, Scott, yes. But that is all.
ASSUME NOTHING!!!!!!
CBE, MBE, MoA and proud of it.
Outta here

Billy Video
DVD Profiler Desktop and Mobile RegistrantStar ContributorTheMadMartian
Alien with an attitude
Registered: March 13, 2007
Reputation: Highest Rating
United States Posts: 13,202
Posted:
PM this userEmail this userView this user's DVD collectionDirect link to this postReply with quote
Quoting movie_madness:
Quote:
DVD Profiler is nice and all but if you're like me, you also have laserdiscs, VHS, Video CDs, DVR recordings, and online movie downloads, and DVD Profiler can't include them.

Actually, DVD Profiler can, it's called custom media types.  Using those, you can track anything you want...even wine.
No dictator, no invader can hold an imprisoned population by force of arms forever.
There is no greater power in the universe than the need for freedom.
Against this power, governments and tyrants and armies cannot stand.
The Centauri learned this lesson once.
We will teach it to them again.
Though it take a thousand years, we will be free.
- Citizen G'Kar
DVD Profiler Unlimited Registrantmovie_madness
Registered: August 7, 2007
Posts: 102
Posted:
PM this userView this user's DVD collectionDirect link to this postReply with quote
Quoting KinoNiki:
Quote:
So you have succeeded in creating another IMDb clone... You're definitely not the first nor the last person to do so, but personally I fail to see what the point is.

DVD Profiler is unique in that it tries to capture what is actually on the disc/film credits. Of course it requires more work than simply scraping data that is already available on the net. But if one doesn't care about the quality or source I guess the latter is acceptable for some people. I use IMDb myself and each tool has its use but I wouldn't like to mix them.

Not an IMDb clone, but something that has the advantage of DVD Profiler, except that it also includes my LDs, VCDs, VHS, etc.  What I did was I imported IMDb's data and linked them to my own collection data, so I can get search results that are actually linked to my collection.  If I search IMDb for something, I get a long list of titles and credits, and they don't tell me if I have them in my collection or not.
 Last edited: by movie_madness
DVD Profiler Desktop and Mobile RegistrantStar ContributorTheMadMartian
Alien with an attitude
Registered: March 13, 2007
Reputation: Highest Rating
United States Posts: 13,202
Posted:
PM this userEmail this userView this user's DVD collectionDirect link to this postReply with quote
Quoting Winston Smith:
Quote:
Locally, Scott, yes. But that is all.

Not true, unless I am misunderstanding you.  Once you create the profile, and upload it, it is visible in your online collecton with all the data that is in the profile...with the exception of cover scans.
No dictator, no invader can hold an imprisoned population by force of arms forever.
There is no greater power in the universe than the need for freedom.
Against this power, governments and tyrants and armies cannot stand.
The Centauri learned this lesson once.
We will teach it to them again.
Though it take a thousand years, we will be free.
- Citizen G'Kar
DVD Profiler Unlimited RegistrantStar ContributorSpikyCactus
I have a Gold Star!
Registered: July 16, 2010
Reputation: High Rating
United Kingdom Posts: 527
Posted:
PM this userVisit this user's homepageView this user's DVD collectionDirect link to this postReply with quote
Quoting TheMadMartian:
Quote:
Quoting movie_madness:
Quote:
DVD Profiler is nice and all but if you're like me, you also have laserdiscs, VHS, Video CDs, DVR recordings, and online movie downloads, and DVD Profiler can't include them.

Actually, DVD Profiler can, it's called custom media types.  Using those, you can track anything you want...even wine.


I've only got about 15 bottles of wine and I tend to drink them anyway, so I'm not sure it's worth bothering. 

Can you use it for cacti?
Do you ever find yourself striving for perfection with an almost worthless attempt at it?  Guttermouth "Lemon Water".  Also, I include in my Profiler database VHS tapes, audio DVDs, audio books (digital, cassette and CD), video games (digital, DVD and CD) and 'enhanced' CDs with video tracks on them, as well as films and TV I've bought digitally.  So I'm an anarchist, deal with it.  Just be thankful I don't include most of my records and CDs etc in it too; don't think I haven't been tempted...
DVD Profiler Unlimited Registrantmovie_madness
Registered: August 7, 2007
Posts: 102
Posted:
PM this userView this user's DVD collectionDirect link to this postReply with quote
Quoting scotthm:
Quote:
Quoting movie_madness:
Quote:
DVD Profiler is nice and all but if you're like me, you also have laserdiscs, VHS, Video CDs, DVR recordings, and online movie downloads, and DVD Profiler can't include them.

Really?  It includes mine.

---------------

But you had to enter all the cast and crew and all other movie details yourself, didn't you?  Even if you create a custom media type for VHS, and even though DVDP's database already has the movie (for the DVD and BD editions), but if you enter a VHS for that movie, you still have to enter all cast and crew and everything else yourself.  You can't "carry over" movie details from one medium to another.  I just can't work with that since I have thousands and thousands of movies.  By importing IMDb's data, I finished cataloguing thousands of my movies in a matter of days.  Now tell me how long you spent on yours.
 Last edited: by movie_madness
DVD Profiler Unlimited Registrantmovie_madness
Registered: August 7, 2007
Posts: 102
Posted:
PM this userView this user's DVD collectionDirect link to this postReply with quote
Quoting TheMadMartian:
Quote:
Quoting movie_madness:
Quote:
DVD Profiler is nice and all but if you're like me, you also have laserdiscs, VHS, Video CDs, DVR recordings, and online movie downloads, and DVD Profiler can't include them.

Actually, DVD Profiler can, it's called custom media types.  Using those, you can track anything you want...even wine.

Hmm, I wonder if Kino or Universal make any wine that comes in wide screen and region-free.  But joking aside, the main flaw I notice in DVD Profiler is that it doesn't have a SEPARATE movie database and a SEPARATE *disc* database.  DVDP seems to use one big database with all the movie and disc details.  If you enter 3 editions of the same movie (say, DVD, BD, HD DVD) into the database, you have to enter the same movie details such as cast and crew 3 different times.  If I create a custom VHS media type like you suggest, I can't enter a VHS edition of a movie and have the movie details "carry over" from an existing DVD or BD edition. 

The technical term used by database professional (which I am one, and I'm sure Ken Cole is too) is that the database is not "normalized", and that leads to a lot of redundant data.
 Last edited: by movie_madness
DVD Profiler Desktop and Mobile RegistrantStar ContributorTheMadMartian
Alien with an attitude
Registered: March 13, 2007
Reputation: Highest Rating
United States Posts: 13,202
Posted:
PM this userEmail this userView this user's DVD collectionDirect link to this postReply with quote
Quoting movie_madness:
Quote:
But you had to enter all the cast and crew and all other movie details yourself, didn't you?  Even if you create a custom media type for VHS, and even though DVDP's database already has the movie (for the DVD and BD editions), but if you enter a VHS for that movie, you still have to enter all cast and crew and everything else yourself.  You can't "carry over" movie details from one medium to another.  I just can't work with that since I have thousands and thousands of movies.

Yes, you can.  I am not at my home computer, so can't check the exact menu selection, but there is a 'Copy DVD/Paste DVD' function that will allow you to copy and paste whatever data you desire into the new profile.

The other option is to simply download an existing profile and convert it to a manual profile, then change the media type.

There is another option, if you like IMDb data, that allows you to import the IMDb cast and crew.  I don't know what the program is called that does this, but I am sure somebody does.

I am sure there are other ways to do it, but the bottom line is, you don't have to enter anything into the new profiles.  In the cases where you do, it shouldn't be more that a couple of clicks.
No dictator, no invader can hold an imprisoned population by force of arms forever.
There is no greater power in the universe than the need for freedom.
Against this power, governments and tyrants and armies cannot stand.
The Centauri learned this lesson once.
We will teach it to them again.
Though it take a thousand years, we will be free.
- Citizen G'Kar
DVD Profiler Unlimited RegistrantStar ContributorWinston Smith
Don't be discommodious
Registered: March 13, 2007
United States Posts: 21,610
Posted:
PM this userEmail this userView this user's DVD collectionDirect link to this postReply with quote
It can even serve as babe profiler. Imagine the ar data, lol. We wont even talk about runtime.

But your custom media data is not contributable. There is info somewhere around here about creating clone profiles.
ASSUME NOTHING!!!!!!
CBE, MBE, MoA and proud of it.
Outta here

Billy Video
DVD Profiler Unlimited Registrantmovie_madness
Registered: August 7, 2007
Posts: 102
Posted:
PM this userView this user's DVD collectionDirect link to this postReply with quote
Quoting KinoNiki:
Quote:
DVD Profiler is unique in that it tries to capture what is actually on the disc/film credits. Of course it requires more work than simply scraping data that is already available on the net. But if one doesn't care about the quality or source I guess the latter is acceptable for some people. I use IMDb myself and each tool has its use but I wouldn't like to mix them.

I've got quite a few DVDP entries of obscure movies that have NO cast and crew info.  Quite a few of my discs aren't even on DVDP's database at all.  It would take me a long time to get all the info, from looking at disc covers and the movies themselves, and manually enter them into the database.  I've done manual entries into DVDP for a while, but I just had to stop.  I've used DVDP for years and it just doesn't have and is never going to have a database large enough to suit my needs. 

OTOH, IMDb has every conceivable movie.  Regarding its accuracy, please believe me when I say that I RARELY see any erroneous entries in the cast and crew.  I'm someone who do pay attention to those things, and naturally I know the movies I own so I'm able to catch errors.  I own mostly old classic movies.  Maybe there are more errors in more recent movies, I dunno.  If you compare a recently entered movie on IMDb to a movie entered a long time ago, naturally the more recently entered movie is likely to have more errors since much fewer people have checked on it.  My observation is that the most frequent IMDb errors occur in sections outside of the cast and crew, such as trivia, awards, etc., which are no concern to me.
DVD Profiler Unlimited Registrantmovie_madness
Registered: August 7, 2007
Posts: 102
Posted:
PM this userView this user's DVD collectionDirect link to this postReply with quote
Quoting TheMadMartian:
Quote:
Yes, you can.  I am not at my home computer, so can't check the exact menu selection, but there is a 'Copy DVD/Paste DVD' function that will allow you to copy and paste whatever data you desire into the new profile.


There is still the problem that a lot of movies are not on any home video medium and thus, not in DVDP's database.  I have thousands of those.  And as I said in my previous posts, a lot of DVDP entries have blank cast and crew too.  The crux of the matter is DVDP just doesn't have a big enough database.  I understand, of course, far fewer people manage DVDP's database than those who manage IMDb.

I stand corrected that you can copy and paste entries.  But here is still the problem that if you make corrections on one, you have to do the same for other entries.
 Last edited: by movie_madness
DVD Profiler Unlimited RegistrantStar ContributorWinston Smith
Don't be discommodious
Registered: March 13, 2007
United States Posts: 21,610
Posted:
PM this userEmail this userView this user's DVD collectionDirect link to this postReply with quote
Ahh theres the rub. Who do you believe for data IMDb or the people who made the film. Ill take the filmmakers everytime. I dont think IMDb, nor mist of its user has ever made or been iinvolved with a film, any film.
ASSUME NOTHING!!!!!!
CBE, MBE, MoA and proud of it.
Outta here

Billy Video
    Invelos Forums->General: General Discussion Page: 1 2 3 ...5  Previous   Next