DonationCoder.com forum
 
Read-only archive. This is a static copy of the DonationCoder.com forum. Reading and searching work as before; logging in, posting and private messages do not.
Archive of 2026-10-21 · About this archive · Main site

Author Topic: IDEA - Create a text doc of an artists discography.  (Read 12,688 times)

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • Posts: 12
IDEA - Create a text doc of an artists discography.
« on: September 06, 2020, 07:36 PM »
This is something I've been trying to do, off-and-on for 12 months - but it's just beyond me.

A program that takes a supplied artist/band and returns a simple text doc (or copyable window) of their discography, ideally scraped from somewhere like allmusic.com (allmusic seems to just list the main entries, rather than somewhere like discogs that shows all variants, releases and pressings).

Shades

  • Member
  • Joined in 2006
  • **
  • Posts: 2,954
Re: IDEA - Create a text doc of an artists discography.
« Reply #1 on: September 06, 2020, 10:08 PM »
What you appear to wish for is a 'scraper' or 'site-scraper'.

While there are enough legitimate use-cases for such software, this kind of software is also used for nefarious purposes (hence no links in my post).
The easy path is usually downhill...

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • Posts: 12
Re: IDEA - Create a text doc of an artists discography.
« Reply #2 on: September 06, 2020, 10:36 PM »
Doesn't have to be a scraper - it's just the first thing that came to mind.

I can't think of an easier method to get a list of information back ..... maybe a direct query of a publicly accessible music database? I think the one used by MusicBrainz Picard might work.

I'm a carpenter, not a coder :D

SleepingWolf

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 118
Re: IDEA - Create a text doc of an artists discography.
« Reply #3 on: November 06, 2020, 09:52 PM »
There is a program by skwire called epCheck that downloads parts of the online database at https://thetvdb.com/, I can't really distinguish why that would be a suitable program, but this suggestion would not. Great idea, fred dred.
-------------------------------------------

Question everything, use a fact-checker.
Respect yourself and respect others.
Peace out!
« Last Edit: November 06, 2020, 10:02 PM by SleepingWolf »

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • Posts: 12
Re: IDEA - Create a text doc of an artists discography.
« Reply #4 on: November 06, 2020, 11:33 PM »
Thanks :)

Maybe using the word 'scrape' was a bad idea?

All I want to do is see what albums I'm missing in my collection, and put that info into a text I can drop into the artists folder - without having to visit allmusic or discogs and laboriously copy and pasting the info line by line.

4wd

  • Pedantic Old Bastard
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,652
    • Coding Snacks
Re: IDEA - Create a text doc of an artists discography.
« Reply #5 on: November 07, 2020, 03:41 AM »
I'm not going to offer an all encompassing solution but here's something simple in PowerShell that uses Discogs - it might inspire someone ;)

Code: ( powershell)
<#
  .\DiscoFile.ps1 <Uri>

  Given the URI from https://www.discogs.com for an artist it will retrieve the
  discography.

  eg.

  .\DiscoFile.ps1 https://www.discogs.com/artist/84752-ACDC

  Will result in a file called 'AC_DC - Discography.txt' in the same directory as
  DiscoFile.ps1
  
#>

param (
  [string]$Uri
)

$web = (Invoke-WebRequest -Uri $Uri).RawContent

$temp = $web -replace '[\s\S]+artist_schema..([\s\S]+?)<\/script>[\s\S]*$', '$1'

$json = $temp | ConvertFrom-Json

$outFile = ($json.name + " - Discography.txt").Split([IO.Path]::GetInvalidFileNameChars()) -join '_'

for ($i = 0; $i -lt $json.album.Count; $i++) {
  "$($json.album[$i].datePublished) - $($json.album[$i].name)" | Out-File -FilePath $outFile -Append
}

Using the example above, output will be to 'AC_DC - Discography.txt' and look like:
Code: (text)
1975 - T.N.T.
1975 - High Voltage
1976 - Dirty Deeds Done Dirt Cheap
1976 - High Voltage
1976 - Dirty Deeds Done Dirt Cheap
1977 - Let There Be Rock
1978 - Powerage
1978 - If You Want Blood You've Got It
1979 - Highway To Hell
1980 - In Concert-215
1980 - Back In Black
1981 - For Those About To Rock (We Salute You)
1983 - Special Radio Interview Disc
1983 - Flick Of The Switch
1985 - Fly On The Wall
1988 - Blow Up Your Video
1990 - AC/DC Interview
1990 - The Razors Edge
1992 - Live

Why did I choose Discogs even though the album data is incomplete?

Code: ( powershell)
$json = $temp | ConvertFrom-Json

Album data is available in JSON format which makes retrieving it trivial.

Having a quick look at MusicBrainz, it appears to also be in JSON format - maybe I'll devote an extra 30 5 mins to this ;)

MusicBrainz version, they don't appear to include the year in the JSON data though:

Code: ( powershell)
<#
  .\DiscoFile-MB.ps1 <Uri>

  Given the URI from https://musicbrainz.org for an artist it will retrieve the
  discography.

  eg.

  .\DiscoFile-MB.ps1 https://musicbrainz.org/artist/66c662b6-6e2f-4930-8610-912e24c63ed1

  Will result in a file called 'AC_DC-MB - Discography.txt' in the same directory as
  DiscoFile.ps1

#>

param (
  [string]$Uri
)

$web = (Invoke-WebRequest -Uri $Uri).RawContent

$temp = $web -replace '[\s\S]+application\/ld\+json..([\s\S]+?)<\/script>[\s\S]*$', '$1'

$json = $temp | ConvertFrom-Json

$outFile = ($json.name + "-MB - Discography.txt").Split([IO.Path]::GetInvalidFileNameChars()) -join '_'

for ($i = 0; $i -lt $json.album.Count; $i++) {
  "$($json.album[$i].name)" | Out-File -FilePath $outFile -Append
}
"I will not follow where the path may lead, but I will go where there is no path, and I will leave a trail." - Muriel Strode, 1903
« Last Edit: November 07, 2020, 03:59 AM by 4wd »

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • Posts: 12
Re: IDEA - Create a text doc of an artists discography.
« Reply #6 on: November 07, 2020, 08:36 AM »
Thank you very, very much for taking the time to do this !

I've never used powershell, so it took half an hour to figure out how to get it to run, but works great now :)

The discogs output is good, but as you said, it's far from complete - so it doesn't really help as a resource to check my collection against.

The MusicBrainz output is brilliant - but no dates means it's a pain to cross reference with my existing files.

But thank you again for taking a look - you've done far more than I ever accomplished in the months and months of tinkering with Python !

Cheers :)



4wd

  • Pedantic Old Bastard
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,652
    • Coding Snacks
Re: IDEA - Create a text doc of an artists discography.
« Reply #7 on: November 07, 2020, 09:21 AM »
Thank you very, very much for taking the time to do this !

No problem, it was quite literally 20 minutes of playing around - working out the RegEx to remove everything I didn't want took the most time.

Quote
The MusicBrainz output is brilliant - but no dates means it's a pain to cross reference with my existing files.

You can get the dates, you just have to follow every album link but they do provide the info in JSON format without having to strip it out of the page.

For example, AC/DC's High Voltage:

XML
JSON

So it can be obtained, just grab the JSON and pull out the first date value, as an example:

Code: ( powershell)
$album = (Invoke-WebRequest -Uri "https://musicbrainz.org/ws/2/release-group/2b81e645-4586-4456-843a-9bc19a217470?inc=releases&fmt=json") | ConvertFrom-Json
"$($album.releases[0].date) - $($album.title)"

The only problem being multiple release dates, eg. for that particular entry there are nine dates ... which one do you choose really?

Considering MusicBrainz provide album data in JSON, I'm kind of surprised that the artist page doesn't provide the same, (I mean including the album details, ie. combined artist+releases).
"I will not follow where the path may lead, but I will go where there is no path, and I will leave a trail." - Muriel Strode, 1903
« Last Edit: November 07, 2020, 09:29 AM by 4wd »

4wd

  • Pedantic Old Bastard
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,652
    • Coding Snacks
Re: IDEA - Create a text doc of an artists discography.
« Reply #8 on: November 07, 2020, 10:40 AM »
This'll just grab the first date in a release:

Code: ( powershell)
<#
  .\DiscoFile-MB2.ps1 <Uri>

  Given the URI from https://musicbrainz.org for an artist it will retrieve the
  discography.

  eg.

  .\DiscoFile-MB2.ps1 https://musicbrainz.org/artist/66c662b6-6e2f-4930-8610-912e24c63ed1

  Will result in a file called 'AC_DC-MB - Discography.txt' in the same directory as
  DiscoFile.ps1

#>

param (
  [string]$Uri
)

$web = (Invoke-WebRequest -Uri $Uri).RawContent

$temp = $web -replace '[\s\S]+application\/ld\+json..([\s\S]+?)<\/script>[\s\S]*$', '$1'

$json = $temp | ConvertFrom-Json

$outFile = ($json.name + "-MB - Discography.txt").Split([IO.Path]::GetInvalidFileNameChars()) -join '_'
if (Test-Path $outFile) {
  Remove-Item $outFile -Force
}

for ($i = 0; $i -lt $json.album.Count; $i++) {
  $albumUri = ($json.album[$i].'@id' + "?inc=releases&fmt=json") -replace '\.org\/', '.org/ws/2/'
  $album = (Invoke-WebRequest -Uri "$($albumUri)") | ConvertFrom-Json
  "$($album.releases[0].date) - $($album.title)" | Out-File -FilePath $outFile -Append
}

Example output:
Code: (text)
1976-02-14 - Dreamboat Annie
1977-04-19 - Magazine
1977-05-15 - Little Queen
1978-10-07 - Dog & Butterfly
1980-02-14 - B??b?? le Strange
1982-06-05 - Private Audition
1983-08-20 - Passionworks
1985-07-06 - Heart
1987-06-05 - Bad Animals
1990-03-26 - Brigade
1993-10-30 - Desire Walks On
2004-06-22 - Jupiters Darling
2010-08-27 - Red Velvet Car
2012-09-28 - Fanatic
2016-07-08 - Beautiful Broken
1980-11-29 - Greatest Hits / Live
1981 - Heart: Best of the Early Years
1995-12-11 - Definitive Collection
1996-07-03 - Ballads: The Greatest Hits
1997-03-11 - These Dreams: Heart???s Greatest Hits
1998-08-25 - Greatest Hits
2000-06-27 - Greatest Hits: 1985???1995
2002-11-26 - The Essential Heart
2005-09-27 - Love Alive
2005-11-15 - The Collection
2006-01-10 - Love Songs
2008 - Playlist: The Very Best of Heart
2009 - Private Audition / Passionworks
2011-06-23 - Heart / Bad Animals
2012-06-01 - Strange Euphoria
2012 - Super Hits
2013-05-13 - Icon
2013 - Original Album Classics
2014-10-14 - The Box Set Series
1991-09-24 - Rock the House Live!
1995-08-09 - The Road Home
2003 - Alive in Seattle
2007-10-23 - Dreamboat Annie Live
2008-08-05 - Live
2014-02-21 - Fanatic Live From Caesars Colosseum
2014-11-07 - Heart & Friends: Home for the Holidays
2016-11-25 - Live at the Royal Albert Hall
2019-01-25 - Live in Atlantic City
1975-06 - Magic Man
1975 - Crazy on You
1975 - How Deep It Goes
1976-12 - Dreamboat Annie
1976 - (Love Me Like Music) I???ll Be Your Song
1977 - Barracuda
1977 - Heartless
1977 - Kick It Out
1977 - Little Queen / Treat Me Well
1978 - Straight On
1978 - Without You
1985-09-14 - Never
1985 - Nothin??? at All
1986-01-18 - These Dreams
1986-07-19 - If Looks Could Kill
1987 - There???s the Girl
1987 - Who Will You Run To
1988-02 - Never / These Dreams
1990-03-28 - All I Wanna Do Is Make Love to You
1990-09 - Stranded
1990 - I Didn???t Want to Need You
1990 - Secret
1991 - You???re the Voice
1993-11 - Black on Black II
1993 - Will You Be There (In the Morning)
1994 - The Woman in Me
1998 - What About Love
2012-08-21 - Fanatic
2013-11-08 - Heart Christmas Single 2013
2017-11-03 - Band on the Run
2017-11-03 - Letting Go
2013-01-15 - Stairway to Heaven (live at the Kennedy Center Honors)
2018-10-23 - Lost Angel
2019-01-11 - B??b?? le Strange
2019-01-16 - Crazy on You
2012-11-12 - A Million Miles (Abe Clements remix)
1988 - If Looks Could Kill
2010-03-05 - WTF+4

You could add filtering to restrict it to certain types of releases by matching in the albumProductionType or albumReleaseType, eg.

Code: ( powershell)
for ($i = 0; $i -lt $json.album.Count; $i++) {
  if ($json.album[$i].albumProductionType -match 'StudioAlbum') {
    $albumUri = ($json.album[$i].'@id' + "?inc=releases&fmt=json") -replace '\.org\/', '.org/ws/2/'
    $album = (Invoke-WebRequest -Uri "$($albumUri)") | ConvertFrom-Json
    "$($album.releases[0].date) - $($album.title)" | Out-File -FilePath $outFile -Append
  }
}

So goes the theory :P
"I will not follow where the path may lead, but I will go where there is no path, and I will leave a trail." - Muriel Strode, 1903
« Last Edit: November 08, 2020, 04:48 AM by 4wd »

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • Posts: 12
Re: IDEA - Create a text doc of an artists discography.
« Reply #9 on: November 07, 2020, 10:43 AM »
(EDIT - I was writing this as you posted your last post above)

The only problem being multiple release dates, eg. for that particular entry there are nine dates ... which one do you choose really?

That's one of the reasons I became fixated on the allmusic site.

Here's the info. There's a nice list of release dates and album titles, surely it can't be that difficult I kept telling myself - to my mind it's just a matter of pulling those bit's of info off the page haha. (My coding skills are extremely basic, and I just couldn't get any of the Python modules to get the info I wanted - or even to work most of the time if I'm honest).

4wd

  • Pedantic Old Bastard
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,652
    • Coding Snacks
Re: IDEA - Create a text doc of an artists discography.
« Reply #10 on: November 07, 2020, 10:46 AM »
That's one of the reasons I became fixated on the allmusic site.

Yes, nicer format but unfortunately in HTML which can be a P.I.T.A to parse.

They might have an API which can be leveraged but I couldn't find one in a quick look at the site.

Addendum: Looks like there is an API through Rovi (TiVo) if you get an API key here - although a quick look seems to indicate no way to get a list of releases for an artist.
"I will not follow where the path may lead, but I will go where there is no path, and I will leave a trail." - Muriel Strode, 1903
« Last Edit: November 07, 2020, 10:56 AM by 4wd »

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • Posts: 12
Re: IDEA - Create a text doc of an artists discography.
« Reply #11 on: November 07, 2020, 02:56 PM »
It makes me very happy that you mention html to be a pain to work with - maybe I'm not as stupid as I thought I was  :D

AFAIR even when I could get beautiful soup or one of the other html parsers to grab the page, there didn't seem to be any of the relevant information in it !

You've given me a lot to play with, thanks again,

Steve.

Nod5

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 1,175
    • Coding Snacks
    • Participated in NANY Challenge Three Times
    • Plugin Coding Medal
Re: IDEA - Create a text doc of an artists discography.
« Reply #12 on: November 07, 2020, 04:11 PM »
Nice 4wd! One idea is to search for artist name and use the first match as artist page URL input into the rest of your program.
The URL format is
Code
https://www.discogs.com/search/?type=artist&title=ac+dc
Other URL parameters can be detected by playing around with options on https://www.discogs.com/search/advanced

This could be a little NANY 2021 app.

4wd

  • Pedantic Old Bastard
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,652
    • Coding Snacks
Re: IDEA - Create a text doc of an artists discography.
« Reply #13 on: November 07, 2020, 10:57 PM »
Nice 4wd!

Thanks!

Quote
One idea is to search for artist name and use the first match as artist page URL input into the rest of your program.
The URL format is
Code
https://www.discogs.com/search/?type=artist&title=ac+dc

I think that given the incomplete records on Discogs, time would be better spent on either MusicBrainz, (filtering their overly complete lists down to just the studio releases, or otherwise selecting what you want), or trying to parse allmusic's HTML, (that'll be fun but it is just one big table, so maybe it won't be that hard - PowerShell can get at elements within HTML, just never really looked at it).

eg. For MusicBrainz, pipe the output into a ListView with checkboxes, select the ones you want, output to a text file - if you want to use manual selection.

Quote
This could be a little NANY 2021 app.

... it might inspire someone ;)

:P

Looking at AllMusic, their info is a little mixed up also, Live albums in amongst Studio albums whereas MusicBrainz splits Studio and Live.

Since MusicBrainz list the Studio albums first, we can prune everything down to just those easily enough:

Code: ( powershell)
<#
  .\DiscoFile-MB3.ps1 <Uri>

  Given the URI from https://musicbrainz.org for an artist it will retrieve the
  discography.

  eg.

  .\DiscoFile-MB3.ps1 https://musicbrainz.org/artist/66c662b6-6e2f-4930-8610-912e24c63ed1

  Will result in a file called 'AC_DC-MB - Discography.txt' in the same directory as
  DiscoFile.ps1

#>

param (
  [string]$Uri
)

$web = (Invoke-WebRequest -Uri $Uri).RawContent

$temp = $web -replace '[\s\S]+application\/ld\+json..([\s\S]+?)<\/script>[\s\S]*$', '$1'

$json = $temp | ConvertFrom-Json

$outFile = ($json.name + "-MB - Discography.txt").Split([IO.Path]::GetInvalidFileNameChars()) -join '_'
if (Test-Path $outFile) {
  Remove-Item $outFile -Force
}

$i = 0
do {
  if ($json.album[$i].albumProductionType -match 'StudioAlbum') {
    $found = $true
    $albumUri = ($json.album[$i].'@id' + "?inc=releases&fmt=json") -replace '\.org\/', '.org/ws/2/'
    $album = (Invoke-WebRequest -Uri "$($albumUri)") | ConvertFrom-Json
    "$($album.releases[0].date) - $($album.title)" | Out-File -FilePath $outFile -Append
  } else {
    $found = $false
  }
  $i++
} while ($found)

Which makes it a little nicer:
Code: (text)
1975-02-17 - High Voltage
1975-12-01 - T.N.T.
1976-05-14 - High Voltage
1976-09-20 - Dirty Deeds Done Dirt Cheap
1977-03-21 - Let There Be Rock
1978-05-05 - Powerage
1979-07-27 - Highway to Hell
1980-07-25 - Back in Black
1981-11-23 - For Those About to Rock (We Salute You)
1983-08-15 - Flick of the Switch
1985-06-28 - Fly on the Wall
1988-01-18 - Blow Up Your Video
1990-09-21 - The Razors Edge
1995-09-22 - Ballbreaker
2000-02-25 - Stiff Upper Lip
2008-10-17 - Black Ice
2014-11-28 - Rock or Bust
2020-11-13 - Power Up

I guess AllMusic could be parsed using the HTML Agility Pack ... if I could work out how to use it ;D
"I will not follow where the path may lead, but I will go where there is no path, and I will leave a trail." - Muriel Strode, 1903
« Last Edit: November 08, 2020, 04:51 AM by 4wd »

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • Posts: 12
Re: IDEA - Create a text doc of an artists discography.
« Reply #14 on: November 08, 2020, 06:58 AM »
Wow.  :Thmbsup:

I'm be happy for the list to contain both studio and live albums, the ones I wouldn't want would be compilations.

Still ..... wow :)

4wd

  • Pedantic Old Bastard
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,652
    • Coding Snacks
Re: IDEA - Create a text doc of an artists discography.
« Reply #15 on: November 08, 2020, 12:53 PM »
Code: ( powershell)
<#
  .\DiscoFile-MB3.ps1 <Uri>

  Given the URI from https://musicbrainz.org for an artist it will retrieve the
  discography.

  eg.

  .\DiscoFile-MB3.ps1 https://musicbrainz.org/artist/66c662b6-6e2f-4930-8610-912e24c63ed1

  Will result in a file called 'AC_DC-MB - Discography.txt' in the same directory as
  DiscoFile-MB3.ps1

#>

param (
  [string]$Uri
)

$web = (Invoke-WebRequest -Uri $Uri).RawContent

$temp = $web -replace '[\s\S]+application\/ld\+json..([\s\S]+?)<\/script>[\s\S]*$', '$1'

$json = $temp | ConvertFrom-Json

$outFile = ($json.name + "-MB - Discography.txt").Split([IO.Path]::GetInvalidFileNameChars()) -join '_'
if (Test-Path $outFile) {
  Remove-Item $outFile -Force
}

for ($i = 0; $i -lt $json.album.Count; $i++) {
  $temp = $json.album[$i].albumReleaseType
  if ($temp -match 'AlbumRelease') {
    $temp = $json.album[$i].albumProductionType
    if (($temp -match '(LiveAlbum)|(StudioAlbum)') -and !($temp -match '(CompilationAlbum)|(SoundtrackAlbum)')) {
      $albumUri = ($json.album[$i].'@id' + "?inc=releases&fmt=json") -replace '\.org\/', '.org/ws/2/'
      $album = (Invoke-WebRequest -Uri "$($albumUri)") | ConvertFrom-Json
      "$($album.releases[0].date.Substring(0,4)) - $($album.title)" | Out-File -FilePath $outFile -Append
    }
  }
}

Get-Content $outFile | Sort-Object | Set-Content $outFile

Code: (text)
1975 - High Voltage
1975 - T.N.T.
1976 - Dirty Deeds Done Dirt Cheap
1976 - High Voltage
1977 - Let There Be Rock
1978 - If You Want Blood You???ve Got It
1978 - Live from the Atlantic Studios
1978 - Powerage
1979 - Highway to Hell
1980 - Back in Black
1981 - For Those About to Rock (We Salute You)
1983 - Flick of the Switch
1985 - Fly on the Wall
1988 - Blow Up Your Video
1990 - The Razors Edge
1992 - Live at Donington
1995 - Ballbreaker
1996 - No Bull: Live - Plaza De Toros, Madrid
1996 - Westwood One Superstars In Concert Series
2000 - Stiff Upper Lip
2001 - Stiff Upper Lip: Live
2007 - Plug Me In
2008 - Black Ice
2011 - Live at River Plate
2014 - Rock or Bust
2018 - Legendary FM Broadcasts - The Paradise Theatre, Boston MA 21st August 1978
2018 - Live At Paradise Theatre Boston 1978
2019 - Live at the Apocalypse
2020 - AC/DC FM Broadcast Boston 1978
2020 - Power Up
"I will not follow where the path may lead, but I will go where there is no path, and I will leave a trail." - Muriel Strode, 1903

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • Posts: 12
Re: IDEA - Create a text doc of an artists discography.
« Reply #16 on: November 08, 2020, 01:17 PM »
:o :D

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • Posts: 12
Re: IDEA - Create a text doc of an artists discography.
« Reply #17 on: November 08, 2020, 02:34 PM »
Any idea why I get

08-11-20 14-17-45.jpg

And my text doc looks like

Code
1975 - High Voltage
1975 - T.N.T.
1976 - Dirty Deeds Done Dirt Cheap
1976 - High Voltage
1977 - Let There Be Rock
1978 - Powerage
1979 - Highway to Hell
1980 - Back in Black
1981 - For Those About to Rock (We Salute You)
1983 - Flick of the Switch
1985 - Fly on the Wall
1988 - Blow Up Your Video
1990 - The Razors Edge
1995 - Ballbreaker
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip


I also get the same errors when searching Aerosmith
Code
.\DiscoFile-MB3.ps1 https://musicbrainz.org/artist/3d2b98e5-556f-4451-a3ff-c50ea18d57cb
Their list looks like

Code
1973 - Aerosmith
1974 - Get Your Wings
1975 - Toys in the Attic
1976 - Rocks
1977 - Draw the Line
1979 - Night in the Ruts
1982 - Rock in a Hard Place
1985 - Done With Mirrors
1987 - Permanent Vacation
1989 - Pump
1993 - Get a Grip
1997 - Nine Lives
2001 - Just Push Play
2004 - Honkin??? on Bobo
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!

It seems to work fine searching for artists with smaller catalogues , ie 3 Doors Down (2386cd66-e923-4e8e-bf14-2eebe2e9b973)

Could it be a 'Too Many Requests' type issue? - they both failed on the 16th request.... coincidence?

(Just for info, I'm logged in on the MB site)
« Last Edit: November 08, 2020, 02:39 PM by fred dred »

4wd

  • Pedantic Old Bastard
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,652
    • Coding Snacks
Re: IDEA - Create a text doc of an artists discography.
« Reply #18 on: November 08, 2020, 10:16 PM »
The only way being logged in, (or not), to the site would matter is if you are also doing searches or browsing at the same time, since you're increasing the number of requests per second.

It looks like they impose a rate limit of one request per second from the same IP, (strange I never hit it), unless it's from specific software, (eg. picard), or uses a recognisable UserAgent, (ie. a browser, PowerShell probably doesn't by default - I might look at how you can add it, if it's possible).

You can try putting in a Start-Sleep -seconds 1 after the for statement, that will delay the requests to about one per second.

Adjust the number up as necessary to increase the delay between requests, especially if you are browsing the site at the same time.

Or use Opera to browse the site with its internal VPN turned on so it looks like a different IP.

EDIT: Looks like PowerShell has its own UserAgent but can use one of the more standard browser settings which may decrease the rate limiting.  I'll look at it later today if I have time, or you can have a play ;)

https://stackoverflow.com/questions/49863488/is-there-a-unique-user-agent-value-for-a-powershell-request
"I will not follow where the path may lead, but I will go where there is no path, and I will leave a trail." - Muriel Strode, 1903
« Last Edit: November 08, 2020, 10:37 PM by 4wd »

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • Posts: 12
Re: IDEA - Create a text doc of an artists discography.
« Reply #19 on: November 09, 2020, 06:59 AM »
I was logged into the MB site, but I wasn't using it - I just added that info because 'more info is more better' when it comes to figuring out why it aint working :)

Adding a delay of 0.55 seconds to the loop seems to have fixed it  :Thmbsup:

4wd

  • Pedantic Old Bastard
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,652
    • Coding Snacks
Re: IDEA - Create a text doc of an artists discography.
« Reply #20 on: November 09, 2020, 08:39 AM »
Now uses Chrome's UserAgent and can set a delay from the command line:

Code: ( powershell)
<#
  .\DiscoFile-MB3.ps1 -uri <Uri> [-delay <seconds>]

  Given the URI from https://musicbrainz.org for an artist it will retrieve the
  discography.

  Where: Uri     = MusicBrainz artist page URI
         seconds = [optional] number of seconds delay between Web Requests (0.1-90)


  eg.

  .\DiscoFile-MB3.ps1 -uri https://musicbrainz.org/artist/66c662b6-6e2f-4930-8610-912e24c63ed1

  .\DiscoFile-MB3.ps1 -uri https://musicbrainz.org/artist/66c662b6-6e2f-4930-8610-912e24c63ed1 -delay 10

  Will result in a file called 'AC_DC-MB - Discography.txt' in the same directory as
  DiscoFile-MB3.ps1

#>
param (
	[Parameter(Mandatory=$true)][string]$Uri,
	[ValidateRange(0.1,90)][decimal]$delay
)
$sleep = $false
if ($delay -gt 0) {
  $sleep = $true
}
$userAgent = [Microsoft.PowerShell.Commands.PSUserAgent]::Chrome

$web = (Invoke-WebRequest -Uri $Uri -UserAgent $userAgent).RawContent

$temp = $web -replace '[\s\S]+application\/ld\+json..([\s\S]+?)<\/script>[\s\S]*$', '$1'
$json = $temp | ConvertFrom-Json

$outFile = ($json.name + "-MB - Discography.txt").Split([IO.Path]::GetInvalidFileNameChars()) -join '_'
if (Test-Path $outFile) {
  Remove-Item $outFile -Force
}

for ($i = 0; $i -lt $json.album.Count; $i++) {
  if ($sleep) {
    Start-Sleep -seconds $delay
  }
  $temp = $json.album[$i].albumReleaseType
  if ($temp -match 'AlbumRelease') {
    $temp = $json.album[$i].albumProductionType
    if (($temp -match '(LiveAlbum)|(StudioAlbum)') -and !($temp -match '(CompilationAlbum)|(SoundtrackAlbum)')) {
      $albumUri = ($json.album[$i].'@id' + "?inc=releases&fmt=json") -replace '\.org\/', '.org/ws/2/'
      $album = (Invoke-WebRequest -Uri "$($albumUri)" -UserAgent $userAgent) | ConvertFrom-Json
      "$($album.releases[0].date.Substring(0,4)) - $($album.title)" | Out-File -FilePath $outFile -Append
    }
  }
}

Get-Content $outFile | Sort-Object | Set-Content $outFile
"I will not follow where the path may lead, but I will go where there is no path, and I will leave a trail." - Muriel Strode, 1903

Original address: https://www.donationcoder.com/forum/index.php?topic=50413.0