Re: programmatic way to fetch latest release for a given major.minor version

Here you go.

Fetches versions and prints most recent minor for each major
Tests all mirrors for speed and prints out the 4 fastest (takes some time)
http://www.crummy.com/software/BeautifulSoup/

Have a nice day !

#! /bin/env python
# -*- coding: utf-8 -*-

import urllib, BeautifulSoup, re, time, sys

def get_all_versions():
soup =
BeautifulSoup.BeautifulSoup( urllib.urlopen( "http://ftp3.fr.postgresql.org/pub/postgresql/source/"
).read() )
for a in soup( 'a', {'href': re.compile( r"v\d+.\d+.\d+" ) } ):
yield map( int, re.search( r"v(\d+)\.(\d+)\.(\d+)*", a['href']
).groups() )

def get_latest_versions():
lastversions = {}
for a,b,c in sorted( get_all_versions() ):
lastversions[ (a,b) ] = c
return sorted( lastversions.items() )

def parse_query_string( url ):
return dict( map( urllib.unquote_plus, pair.split('=',1) ) for pair in
re.split( "&(?:amp;|)", urllib.splitquery( url )[1] ) )

def get_mirrors():
soup =
BeautifulSoup.BeautifulSoup( urllib.urlopen( "http://wwwmaster.postgresql.org/download/mirrors-ftp"
).read() )
for a in soup( 'a', {'href': re.compile( r"\?setmir=.*url=" ) } ):
yield parse_query_string( a['href'] )['url']

def get_fastest_mirrors( urls, filename ):
for url in urls:
sys.stdout.write( " %s\r" % url )
t = time.time()
try:
urllib.urlopen( url + filename )
except:
pass
d = time.time()-t
print "%.02f s" % d
yield d, url

for major, minor in get_latest_versions():
print "%d.%d.%d" % (major[0], major[1], minor)

mirrors = get_mirrors()
fastest = sorted( get_fastest_mirrors( mirrors, "sync_timestamp" ))[:4]
for d, mirror in fastest:
print "%.02f s %s" % (d,mirror)

On Tue, 10 Apr 2007 00:34:02 +0200, Andrew Hammond
<andrew(dot)george(dot)hammond(at)gmail(dot)com> wrote:

> On 4/9/07, CAJ CAJ <pguser(at)gmail(dot)com> wrote:
>> On 9 Apr 2007 14:47:20 -0700, Andrew Hammond
>> <andrew(dot)george(dot)hammond(at)gmail(dot)com> wrote:
>> > I'm writing a script that wants to know the latest release for a given
>> > major.minor version. Is there some better way than parsing
>> > http://www.postgresql.org/ftp/source/ or trying to
>> connect to ftp
>> > (which is invariably timing out on me today. Is that box getting
>> > hammered or something?) and doing the parsing that? Both approaches
>> > feel quite awkward to me.
>>
>> Use wget to download via HTTP (added recently). Probably wise to add a
>> couple mirrors in your script.
>
> I'm not asking how to download stuff. I'm asking how to figure out the
> current release number for a given major.minor. I thought that was
> clear in my original post, but I guess not. For example, how do I
> determine (programmatically) the lastest version of 8.1.
>
> I'm also interested in a clever way to select one of the "close"
> mirrors at random for downloading via http. However I had planned to
> hold that question until I'd solved the first issue.
>
> Andrew
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq