Topic: Collaborations Archive: Wayback Machine to the rescue?

[Ed Wenn]

I really have no idea why I didn't think of this earlier (and apologies if anyone mentioned this already), but I just checked the Wayback Machine and there's a pretty decent backup of the old Collaborations galleries online from sometime in 2015 

This is great news because it means that we can grab the 'lost' images and the text from there and re-post them on our new Collaborations page.

Take a look: https://web.archive.org/web/20150618160824/http://www.filmwasters.com/collaborations/v/?g2_GALLERYSID=99442c169be52656d22ef7bbfd12a4b9

Now, all I need is some helpers and I'll be able to get this going. If anyone has any bright ideas about how to pull down the whole thing in one go then I'm listening, but otherwise it would be cool if we could each take responsibility for a couple of galleries and download the images and text, save them to a ZIP and then mail them to filmwasters [at] gmail.com with an appropriate title.

It makes the most sense to work from the start I guess.

If anyone's interested in helping out with this, then please let me know by replying to this and we'll divvy up the workload according to how many people have some time to give.

Cheers in advance.

[Francois]

Easiest would be to rip it in one go using HTTrack....
https://www.httrack.com/

[Ed Wenn]

Cool. I like 'easiest'. Let me take a look. Thanks, Francois 

[Francois]

Best part is that it comes in a portable version. No install required.

[Ed Wenn]

Except I couldn't get it to work against the Internet Archive. Boo! Doesn't mean it won't work if someone who knows what they're doing is driving the tool, but I ran out of patience with it.

In the meantime, however, I found a tool on GitHub that does the job (https://github.com/hartator/wayback-machine-downloader) - even though it runs in Ruby, which I needed to install. Good news, is that I now have an offline copy of the old Collabs, but the bad news is that it's not static Web pages, but all of the original PHP stuff. Which is a pain, but to be expected.

I then found another tool on GitHub which can, apparently, convert PHP into static HTML (https://github.com/neurobin/php2html), so I'm giving that a go in a few minutes. Only issue is that it runs in Python which I didn't have....so back to the install process we go.
 

Fingers crossed this works....but even if it does, that just gets me an offline archive in static HTML.I still have to upload the damn things to the new site.  If only I still drank coffee...and didn't have a family, a job and too many other hobbies 

[Francois]

Do you think you could extract everything from a PDF file?
I have Acrobat on my USB stick and it has a site ripping feature.

[Francois]

I've been working on this puzzle and I think I'm getting somewhere!
I'm using the following link https://web.archive.org/web/20150618160824im_/http://www.filmwasters.com/collaborations/v/?g2_GALLERYSID=99442c169be52656d22ef7bbfd12a4b9

Notice the im_ after the date. This hack hides the Wayback machine's header.
I'm now ripping the site page by page using Acrobat...
Then, I'll convert it back to HTML and see what comes out of it...

[Francois]

This is harder than it seems because you get a whole bunch of everything and it's a big mess.
But I can download one collab at a time pretty easily with just a few clicks and operations.
But it's going to be long with 37 collabs on tap...

[Nigel]

Let me know if you need help - I'll clarify that by saying I know almost nothing about Python or Ruby, but I'm a whizz at cutting and pasting! 

[Nigel]

Have you seen this: https://www.waybackmachinedownloader.com/en/

They say you can download a site, up to 20,000 pages (that should do it!) for $15. Sounds like a bargain.

I tried the preview of the demo, which was just three pages and they looked like the original. They give installation advice here: https://www.waybackmachinedownloader.com/en/how-to-download-site-from-wayback-machine/

It's still processing. I'll stick the download of the demo pages in my dropbox and share them once it's cooked. If they work it seems like the answer. I'm happy to chip in the 15 bucks.

[Nigel]

See if this link for the demo works for you: mo/filmwasters.com_nigel.rumsey_gmail.com_iwn7j4xyzm0/

As of 21:05 UK time it's still 'scraping'.

[Francois]

I managed to do something similar using HTTTrack but it has the annoying tendency to replicate the structure of the wayback machine and not the original website. This makes separating the images and text a pain.
But the Acrobat solution I found gets me a single page and the related images only. But the problem is that it's a manual, slow and slightly labor intensive process that spews out a bunch of PNG files and their accompanying HTML file. But at least the results are "simple" to access.

[Nigel]

This is a link to the download sample. It says you need to use their instructions to install. I don't think there's anything special about it they're just advising people it's not going to render properly if you open it in a browser.

https://www.dropbox.com/s/esoshiq35800o6k/filmwasters.zip?dl=0

If this works, it seems like the way to go.

[Francois]

Yeah, it does create a structure a bit like the one from HTTrack...
Here's what mine looks like for collab 36
https://drive.google.com/open?id=1DYxfnqiRDeSGBMTTFNS6iIk0qkE0Jea8

[Nigel]

Yeah, it does create a structure a bit like the one from HTTrack...
Here's what mine looks like for collab 36
https://drive.google.com/open?id=1DYxfnqiRDeSGBMTTFNS6iIk0qkE0Jea8

That works as well. I guess it just depends which is easiest for Ed to get to work on the site.

[Andrea.]

Please remind me; what language are you speaking :-)

[Francois]

 
I guess I could call it geeklish with a strong HTML accent