topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Sunday December 5, 2021, 7:45 am
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: IDEA - Create a text doc of an artists discography.  (Read 3143 times)

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • default avatar
  • Posts: 12
    • View Profile
    • Donate to Member
IDEA - Create a text doc of an artists discography.
« on: September 06, 2020, 02:36 PM »
This is something I've been trying to do, off-and-on for 12 months - but it's just beyond me.

A program that takes a supplied artist/band and returns a simple text doc (or copyable window) of their discography, ideally scraped from somewhere like allmusic.com (allmusic seems to just list the main entries, rather than somewhere like discogs that shows all variants, releases and pressings).


Shades

  • Member
  • Joined in 2006
  • **
  • Posts: 2,849
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #1 on: September 06, 2020, 05:08 PM »
What you appear to wish for is a 'scraper' or 'site-scraper'.

While there are enough legitimate use-cases for such software, this kind of software is also used for nefarious purposes (hence no links in my post).

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • default avatar
  • Posts: 12
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #2 on: September 06, 2020, 05:36 PM »
Doesn't have to be a scraper - it's just the first thing that came to mind.

I can't think of an easier method to get a list of information back ..... maybe a direct query of a publicly accessible music database? I think the one used by MusicBrainz Picard might work.

I'm a carpenter, not a coder :D

SleepingWolf

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 118
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #3 on: November 06, 2020, 03:52 PM »
There is a program by skwire called epCheck that downloads parts of the online database at https://thetvdb.com/, I can't really distinguish why that would be a suitable program, but this suggestion would not. Great idea, fred dred.
-------------------------------------------

Question everything, use a fact-checker.
Respect yourself and respect others.
Peace out!
« Last Edit: November 06, 2020, 04:02 PM by SleepingWolf »

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • default avatar
  • Posts: 12
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #4 on: November 06, 2020, 05:33 PM »
Thanks :)

Maybe using the word 'scrape' was a bad idea?

All I want to do is see what albums I'm missing in my collection, and put that info into a text I can drop into the artists folder - without having to visit allmusic or discogs and laboriously copy and pasting the info line by line.

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,538
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #5 on: November 06, 2020, 09:41 PM »
I'm not going to offer an all encompassing solution but here's something simple in PowerShell that uses Discogs - it might inspire someone ;)

Code: PowerShell [Select]
  1. <#
  2.   .\DiscoFile.ps1 <Uri>
  3.  
  4.   Given the URI from https://www.discogs.com for an artist it will retrieve the
  5.   discography.
  6.  
  7.   eg.
  8.  
  9.   .\DiscoFile.ps1 https://www.discogs.com/artist/84752-ACDC
  10.  
  11.   Will result in a file called 'AC_DC - Discography.txt' in the same directory as
  12.   DiscoFile.ps1
  13.  
  14. #>
  15.  
  16. param (
  17.   [string]$Uri
  18. )
  19.  
  20. $web = (Invoke-WebRequest -Uri $Uri).RawContent
  21.  
  22. $temp = $web -replace '[\s\S]+artist_schema..([\s\S]+?)<\/script>[\s\S]*$', '$1'
  23.  
  24. $json = $temp | ConvertFrom-Json
  25.  
  26. $outFile = ($json.name + " - Discography.txt").Split([IO.Path]::GetInvalidFileNameChars()) -join '_'
  27.  
  28. for ($i = 0; $i -lt $json.album.Count; $i++) {
  29.   "$($json.album[$i].datePublished) - $($json.album[$i].name)" | Out-File -FilePath $outFile -Append
  30. }

Using the example above, output will be to 'AC_DC - Discography.txt' and look like:
Code: Text [Select]
  1. 1975 - T.N.T.
  2. 1975 - High Voltage
  3. 1976 - Dirty Deeds Done Dirt Cheap
  4. 1976 - High Voltage
  5. 1976 - Dirty Deeds Done Dirt Cheap
  6. 1977 - Let There Be Rock
  7. 1978 - Powerage
  8. 1978 - If You Want Blood You've Got It
  9. 1979 - Highway To Hell
  10. 1980 - In Concert-215
  11. 1980 - Back In Black
  12. 1981 - For Those About To Rock (We Salute You)
  13. 1983 - Special Radio Interview Disc
  14. 1983 - Flick Of The Switch
  15. 1985 - Fly On The Wall
  16. 1988 - Blow Up Your Video
  17. 1990 - AC/DC Interview
  18. 1990 - The Razors Edge
  19. 1992 - Live

Why did I choose Discogs even though the album data is incomplete?

Code: PowerShell [Select]
  1. $json = $temp | ConvertFrom-Json

Album data is available in JSON format which makes retrieving it trivial.

Having a quick look at MusicBrainz, it appears to also be in JSON format - maybe I'll devote an extra 30 5 mins to this ;)

MusicBrainz version, they don't appear to include the year in the JSON data though:

Code: PowerShell [Select]
  1. <#
  2.   .\DiscoFile-MB.ps1 <Uri>
  3.  
  4.   Given the URI from https://musicbrainz.org for an artist it will retrieve the
  5.   discography.
  6.  
  7.   eg.
  8.  
  9.   .\DiscoFile-MB.ps1 https://musicbrainz.org/artist/66c662b6-6e2f-4930-8610-912e24c63ed1
  10.  
  11.   Will result in a file called 'AC_DC-MB - Discography.txt' in the same directory as
  12.   DiscoFile.ps1
  13.  
  14. #>
  15.  
  16. param (
  17.   [string]$Uri
  18. )
  19.  
  20. $web = (Invoke-WebRequest -Uri $Uri).RawContent
  21.  
  22. $temp = $web -replace '[\s\S]+application\/ld\+json..([\s\S]+?)<\/script>[\s\S]*$', '$1'
  23.  
  24. $json = $temp | ConvertFrom-Json
  25.  
  26. $outFile = ($json.name + "-MB - Discography.txt").Split([IO.Path]::GetInvalidFileNameChars()) -join '_'
  27.  
  28. for ($i = 0; $i -lt $json.album.Count; $i++) {
  29.   "$($json.album[$i].name)" | Out-File -FilePath $outFile -Append
  30. }
« Last Edit: November 06, 2020, 09:59 PM by 4wd »

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • default avatar
  • Posts: 12
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #6 on: November 07, 2020, 02:36 AM »
Thank you very, very much for taking the time to do this !

I've never used powershell, so it took half an hour to figure out how to get it to run, but works great now :)

The discogs output is good, but as you said, it's far from complete - so it doesn't really help as a resource to check my collection against.

The MusicBrainz output is brilliant - but no dates means it's a pain to cross reference with my existing files.

But thank you again for taking a look - you've done far more than I ever accomplished in the months and months of tinkering with Python !

Cheers :)

 


4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,538
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #7 on: November 07, 2020, 03:21 AM »
Thank you very, very much for taking the time to do this !

No problem, it was quite literally 20 minutes of playing around - working out the RegEx to remove everything I didn't want took the most time.

The MusicBrainz output is brilliant - but no dates means it's a pain to cross reference with my existing files.

You can get the dates, you just have to follow every album link but they do provide the info in JSON format without having to strip it out of the page.

For example, AC/DC's High Voltage:

XML
JSON

So it can be obtained, just grab the JSON and pull out the first date value, as an example:

Code: PowerShell [Select]
  1. $album = (Invoke-WebRequest -Uri "https://musicbrainz.org/ws/2/release-group/2b81e645-4586-4456-843a-9bc19a217470?inc=releases&fmt=json") | ConvertFrom-Json
  2. "$($album.releases[0].date) - $($album.title)"

The only problem being multiple release dates, eg. for that particular entry there are nine dates ... which one do you choose really?

Considering MusicBrainz provide album data in JSON, I'm kind of surprised that the artist page doesn't provide the same, (I mean including the album details, ie. combined artist+releases).
« Last Edit: November 07, 2020, 03:29 AM by 4wd »

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,538
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #8 on: November 07, 2020, 04:40 AM »
This'll just grab the first date in a release:

Code: PowerShell [Select]
  1. <#
  2.   .\DiscoFile-MB2.ps1 <Uri>
  3.  
  4.   Given the URI from https://musicbrainz.org for an artist it will retrieve the
  5.   discography.
  6.  
  7.   eg.
  8.  
  9.   .\DiscoFile-MB2.ps1 https://musicbrainz.org/artist/66c662b6-6e2f-4930-8610-912e24c63ed1
  10.  
  11.   Will result in a file called 'AC_DC-MB - Discography.txt' in the same directory as
  12.   DiscoFile.ps1
  13.  
  14. #>
  15.  
  16. param (
  17.   [string]$Uri
  18. )
  19.  
  20. $web = (Invoke-WebRequest -Uri $Uri).RawContent
  21.  
  22. $temp = $web -replace '[\s\S]+application\/ld\+json..([\s\S]+?)<\/script>[\s\S]*$', '$1'
  23.  
  24. $json = $temp | ConvertFrom-Json
  25.  
  26. $outFile = ($json.name + "-MB - Discography.txt").Split([IO.Path]::GetInvalidFileNameChars()) -join '_'
  27. if (Test-Path $outFile) {
  28.   Remove-Item $outFile -Force
  29. }
  30.  
  31. for ($i = 0; $i -lt $json.album.Count; $i++) {
  32.   $albumUri = ($json.album[$i].'@id' + "?inc=releases&fmt=json") -replace '\.org\/', '.org/ws/2/'
  33.   $album = (Invoke-WebRequest -Uri "$($albumUri)") | ConvertFrom-Json
  34.   "$($album.releases[0].date) - $($album.title)" | Out-File -FilePath $outFile -Append
  35. }

Example output:
Code: Text [Select]
  1. 1976-02-14 - Dreamboat Annie
  2. 1977-04-19 - Magazine
  3. 1977-05-15 - Little Queen
  4. 1978-10-07 - Dog & Butterfly
  5. 1980-02-14 - B??b?? le Strange
  6. 1982-06-05 - Private Audition
  7. 1983-08-20 - Passionworks
  8. 1985-07-06 - Heart
  9. 1987-06-05 - Bad Animals
  10. 1990-03-26 - Brigade
  11. 1993-10-30 - Desire Walks On
  12. 2004-06-22 - Jupiters Darling
  13. 2010-08-27 - Red Velvet Car
  14. 2012-09-28 - Fanatic
  15. 2016-07-08 - Beautiful Broken
  16. 1980-11-29 - Greatest Hits / Live
  17. 1981 - Heart: Best of the Early Years
  18. 1995-12-11 - Definitive Collection
  19. 1996-07-03 - Ballads: The Greatest Hits
  20. 1997-03-11 - These Dreams: Heart???s Greatest Hits
  21. 1998-08-25 - Greatest Hits
  22. 2000-06-27 - Greatest Hits: 1985???1995
  23. 2002-11-26 - The Essential Heart
  24. 2005-09-27 - Love Alive
  25. 2005-11-15 - The Collection
  26. 2006-01-10 - Love Songs
  27. 2008 - Playlist: The Very Best of Heart
  28. 2009 - Private Audition / Passionworks
  29. 2011-06-23 - Heart / Bad Animals
  30. 2012-06-01 - Strange Euphoria
  31. 2012 - Super Hits
  32. 2013-05-13 - Icon
  33. 2013 - Original Album Classics
  34. 2014-10-14 - The Box Set Series
  35. 1991-09-24 - Rock the House Live!
  36. 1995-08-09 - The Road Home
  37. 2003 - Alive in Seattle
  38. 2007-10-23 - Dreamboat Annie Live
  39. 2008-08-05 - Live
  40. 2014-02-21 - Fanatic Live From Caesars Colosseum
  41. 2014-11-07 - Heart & Friends: Home for the Holidays
  42. 2016-11-25 - Live at the Royal Albert Hall
  43. 2019-01-25 - Live in Atlantic City
  44. 1975-06 - Magic Man
  45. 1975 - Crazy on You
  46. 1975 - How Deep It Goes
  47. 1976-12 - Dreamboat Annie
  48. 1976 - (Love Me Like Music) I???ll Be Your Song
  49. 1977 - Barracuda
  50. 1977 - Heartless
  51. 1977 - Kick It Out
  52. 1977 - Little Queen / Treat Me Well
  53. 1978 - Straight On
  54. 1978 - Without You
  55. 1985-09-14 - Never
  56. 1985 - Nothin??? at All
  57. 1986-01-18 - These Dreams
  58. 1986-07-19 - If Looks Could Kill
  59. 1987 - There???s the Girl
  60. 1987 - Who Will You Run To
  61. 1988-02 - Never / These Dreams
  62. 1990-03-28 - All I Wanna Do Is Make Love to You
  63. 1990-09 - Stranded
  64. 1990 - I Didn???t Want to Need You
  65. 1990 - Secret
  66. 1991 - You???re the Voice
  67. 1993-11 - Black on Black II
  68. 1993 - Will You Be There (In the Morning)
  69. 1994 - The Woman in Me
  70. 1998 - What About Love
  71. 2012-08-21 - Fanatic
  72. 2013-11-08 - Heart Christmas Single 2013
  73. 2017-11-03 - Band on the Run
  74. 2017-11-03 - Letting Go
  75. 2013-01-15 - Stairway to Heaven (live at the Kennedy Center Honors)
  76. 2018-10-23 - Lost Angel
  77. 2019-01-11 - B??b?? le Strange
  78. 2019-01-16 - Crazy on You
  79. 2012-11-12 - A Million Miles (Abe Clements remix)
  80. 1988 - If Looks Could Kill
  81. 2010-03-05 - WTF+4

You could add filtering to restrict it to certain types of releases by matching in the albumProductionType or albumReleaseType, eg.

Code: PowerShell [Select]
  1. for ($i = 0; $i -lt $json.album.Count; $i++) {
  2.   if ($json.album[$i].albumProductionType -match 'StudioAlbum') {
  3.     $albumUri = ($json.album[$i].'@id' + "?inc=releases&fmt=json") -replace '\.org\/', '.org/ws/2/'
  4.     $album = (Invoke-WebRequest -Uri "$($albumUri)") | ConvertFrom-Json
  5.     "$($album.releases[0].date) - $($album.title)" | Out-File -FilePath $outFile -Append
  6.   }
  7. }

So goes the theory :P
« Last Edit: November 07, 2020, 10:48 PM by 4wd »

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • default avatar
  • Posts: 12
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #9 on: November 07, 2020, 04:43 AM »
(EDIT - I was writing this as you posted your last post above)

The only problem being multiple release dates, eg. for that particular entry there are nine dates ... which one do you choose really?

That's one of the reasons I became fixated on the allmusic site.

Here's the info. There's a nice list of release dates and album titles, surely it can't be that difficult I kept telling myself - to my mind it's just a matter of pulling those bit's of info off the page haha. (My coding skills are extremely basic, and I just couldn't get any of the Python modules to get the info I wanted - or even to work most of the time if I'm honest).


4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,538
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #10 on: November 07, 2020, 04:46 AM »
That's one of the reasons I became fixated on the allmusic site.

Yes, nicer format but unfortunately in HTML which can be a P.I.T.A to parse.

They might have an API which can be leveraged but I couldn't find one in a quick look at the site.

Addendum: Looks like there is an API through Rovi (TiVo) if you get an API key here - although a quick look seems to indicate no way to get a list of releases for an artist.
« Last Edit: November 07, 2020, 04:56 AM by 4wd »

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • default avatar
  • Posts: 12
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #11 on: November 07, 2020, 08:56 AM »
It makes me very happy that you mention html to be a pain to work with - maybe I'm not as stupid as I thought I was  :D

AFAIR even when I could get beautiful soup or one of the other html parsers to grab the page, there didn't seem to be any of the relevant information in it !

You've given me a lot to play with, thanks again,

Steve.


Nod5

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 1,129
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #12 on: November 07, 2020, 10:11 AM »
Nice 4wd! One idea is to search for artist name and use the first match as artist page URL input into the rest of your program.
The URL format is
https://www.discogs.com/search/?type=artist&title=ac+dc
Other URL parameters can be detected by playing around with options on https://www.discogs.com/search/advanced

This could be a little NANY 2021 app.

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,538
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #13 on: November 07, 2020, 04:57 PM »
Nice 4wd!

Thanks!

One idea is to search for artist name and use the first match as artist page URL input into the rest of your program.
The URL format is
https://www.discogs.com/search/?type=artist&title=ac+dc

I think that given the incomplete records on Discogs, time would be better spent on either MusicBrainz, (filtering their overly complete lists down to just the studio releases, or otherwise selecting what you want), or trying to parse allmusic's HTML, (that'll be fun but it is just one big table, so maybe it won't be that hard - PowerShell can get at elements within HTML, just never really looked at it).

eg. For MusicBrainz, pipe the output into a ListView with checkboxes, select the ones you want, output to a text file - if you want to use manual selection.

This could be a little NANY 2021 app.

... it might inspire someone ;)

 :P

Looking at AllMusic, their info is a little mixed up also, Live albums in amongst Studio albums whereas MusicBrainz splits Studio and Live.

Since MusicBrainz list the Studio albums first, we can prune everything down to just those easily enough:

Code: PowerShell [Select]
  1. <#
  2.   .\DiscoFile-MB3.ps1 <Uri>
  3.  
  4.   Given the URI from https://musicbrainz.org for an artist it will retrieve the
  5.   discography.
  6.  
  7.   eg.
  8.  
  9.   .\DiscoFile-MB3.ps1 https://musicbrainz.org/artist/66c662b6-6e2f-4930-8610-912e24c63ed1
  10.  
  11.   Will result in a file called 'AC_DC-MB - Discography.txt' in the same directory as
  12.   DiscoFile.ps1
  13.  
  14. #>
  15.  
  16. param (
  17.   [string]$Uri
  18. )
  19.  
  20. $web = (Invoke-WebRequest -Uri $Uri).RawContent
  21.  
  22. $temp = $web -replace '[\s\S]+application\/ld\+json..([\s\S]+?)<\/script>[\s\S]*$', '$1'
  23.  
  24. $json = $temp | ConvertFrom-Json
  25.  
  26. $outFile = ($json.name + "-MB - Discography.txt").Split([IO.Path]::GetInvalidFileNameChars()) -join '_'
  27. if (Test-Path $outFile) {
  28.   Remove-Item $outFile -Force
  29. }
  30.  
  31. $i = 0
  32. do {
  33.   if ($json.album[$i].albumProductionType -match 'StudioAlbum') {
  34.     $found = $true
  35.     $albumUri = ($json.album[$i].'@id' + "?inc=releases&fmt=json") -replace '\.org\/', '.org/ws/2/'
  36.     $album = (Invoke-WebRequest -Uri "$($albumUri)") | ConvertFrom-Json
  37.     "$($album.releases[0].date) - $($album.title)" | Out-File -FilePath $outFile -Append
  38.   } else {
  39.     $found = $false
  40.   }
  41.   $i++
  42. } while ($found)

Which makes it a little nicer:
Code: Text [Select]
  1. 1975-02-17 - High Voltage
  2. 1975-12-01 - T.N.T.
  3. 1976-05-14 - High Voltage
  4. 1976-09-20 - Dirty Deeds Done Dirt Cheap
  5. 1977-03-21 - Let There Be Rock
  6. 1978-05-05 - Powerage
  7. 1979-07-27 - Highway to Hell
  8. 1980-07-25 - Back in Black
  9. 1981-11-23 - For Those About to Rock (We Salute You)
  10. 1983-08-15 - Flick of the Switch
  11. 1985-06-28 - Fly on the Wall
  12. 1988-01-18 - Blow Up Your Video
  13. 1990-09-21 - The Razors Edge
  14. 1995-09-22 - Ballbreaker
  15. 2000-02-25 - Stiff Upper Lip
  16. 2008-10-17 - Black Ice
  17. 2014-11-28 - Rock or Bust
  18. 2020-11-13 - Power Up

I guess AllMusic could be parsed using the HTML Agility Pack ... if I could work out how to use it ;D
« Last Edit: November 07, 2020, 10:51 PM by 4wd »

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • default avatar
  • Posts: 12
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #14 on: November 08, 2020, 12:58 AM »
Wow.  :Thmbsup:

I'm be happy for the list to contain both studio and live albums, the ones I wouldn't want would be compilations.

Still ..... wow :)

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,538
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #15 on: November 08, 2020, 06:53 AM »
Code: PowerShell [Select]
  1. <#
  2.   .\DiscoFile-MB3.ps1 <Uri>
  3.  
  4.   Given the URI from https://musicbrainz.org for an artist it will retrieve the
  5.   discography.
  6.  
  7.   eg.
  8.  
  9.   .\DiscoFile-MB3.ps1 https://musicbrainz.org/artist/66c662b6-6e2f-4930-8610-912e24c63ed1
  10.  
  11.   Will result in a file called 'AC_DC-MB - Discography.txt' in the same directory as
  12.   DiscoFile-MB3.ps1
  13.  
  14. #>
  15.  
  16. param (
  17.   [string]$Uri
  18. )
  19.  
  20. $web = (Invoke-WebRequest -Uri $Uri).RawContent
  21.  
  22. $temp = $web -replace '[\s\S]+application\/ld\+json..([\s\S]+?)<\/script>[\s\S]*$', '$1'
  23.  
  24. $json = $temp | ConvertFrom-Json
  25.  
  26. $outFile = ($json.name + "-MB - Discography.txt").Split([IO.Path]::GetInvalidFileNameChars()) -join '_'
  27. if (Test-Path $outFile) {
  28.   Remove-Item $outFile -Force
  29. }
  30.  
  31. for ($i = 0; $i -lt $json.album.Count; $i++) {
  32.   $temp = $json.album[$i].albumReleaseType
  33.   if ($temp -match 'AlbumRelease') {
  34.     $temp = $json.album[$i].albumProductionType
  35.     if (($temp -match '(LiveAlbum)|(StudioAlbum)') -and !($temp -match '(CompilationAlbum)|(SoundtrackAlbum)')) {
  36.       $albumUri = ($json.album[$i].'@id' + "?inc=releases&fmt=json") -replace '\.org\/', '.org/ws/2/'
  37.       $album = (Invoke-WebRequest -Uri "$($albumUri)") | ConvertFrom-Json
  38.       "$($album.releases[0].date.Substring(0,4)) - $($album.title)" | Out-File -FilePath $outFile -Append
  39.     }
  40.   }
  41. }
  42.  
  43. Get-Content $outFile | Sort-Object | Set-Content $outFile

Code: Text [Select]
  1. 1975 - High Voltage
  2. 1975 - T.N.T.
  3. 1976 - Dirty Deeds Done Dirt Cheap
  4. 1976 - High Voltage
  5. 1977 - Let There Be Rock
  6. 1978 - If You Want Blood You???ve Got It
  7. 1978 - Live from the Atlantic Studios
  8. 1978 - Powerage
  9. 1979 - Highway to Hell
  10. 1980 - Back in Black
  11. 1981 - For Those About to Rock (We Salute You)
  12. 1983 - Flick of the Switch
  13. 1985 - Fly on the Wall
  14. 1988 - Blow Up Your Video
  15. 1990 - The Razors Edge
  16. 1992 - Live at Donington
  17. 1995 - Ballbreaker
  18. 1996 - No Bull: Live - Plaza De Toros, Madrid
  19. 1996 - Westwood One Superstars In Concert Series
  20. 2000 - Stiff Upper Lip
  21. 2001 - Stiff Upper Lip: Live
  22. 2007 - Plug Me In
  23. 2008 - Black Ice
  24. 2011 - Live at River Plate
  25. 2014 - Rock or Bust
  26. 2018 - Legendary FM Broadcasts - The Paradise Theatre, Boston MA 21st August 1978
  27. 2018 - Live At Paradise Theatre Boston 1978
  28. 2019 - Live at the Apocalypse
  29. 2020 - AC/DC FM Broadcast Boston 1978
  30. 2020 - Power Up

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • default avatar
  • Posts: 12
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #16 on: November 08, 2020, 07:17 AM »
 :o :D

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • default avatar
  • Posts: 12
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #17 on: November 08, 2020, 08:34 AM »
Any idea why I get

08-11-20 14-17-45.jpgIDEA - Create a text doc of an artists discography.

And my text doc looks like

1975 - High Voltage
1975 - T.N.T.
1976 - Dirty Deeds Done Dirt Cheap
1976 - High Voltage
1977 - Let There Be Rock
1978 - Powerage
1979 - Highway to Hell
1980 - Back in Black
1981 - For Those About to Rock (We Salute You)
1983 - Flick of the Switch
1985 - Fly on the Wall
1988 - Blow Up Your Video
1990 - The Razors Edge
1995 - Ballbreaker
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip
2000 - Stiff Upper Lip


I also get the same errors when searching Aerosmith
.\DiscoFile-MB3.ps1 https://musicbrainz.org/artist/3d2b98e5-556f-4451-a3ff-c50ea18d57cb
Their list looks like

1973 - Aerosmith
1974 - Get Your Wings
1975 - Toys in the Attic
1976 - Rocks
1977 - Draw the Line
1979 - Night in the Ruts
1982 - Rock in a Hard Place
1985 - Done With Mirrors
1987 - Permanent Vacation
1989 - Pump
1993 - Get a Grip
1997 - Nine Lives
2001 - Just Push Play
2004 - Honkin??? on Bobo
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!
2012 - Music From Another Dimension!

It seems to work fine searching for artists with smaller catalogues , ie 3 Doors Down (2386cd66-e923-4e8e-bf14-2eebe2e9b973)

Could it be a 'Too Many Requests' type issue? - they both failed on the 16th request.... coincidence?

(Just for info, I'm logged in on the MB site)
« Last Edit: November 08, 2020, 08:39 AM by fred dred »

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,538
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #18 on: November 08, 2020, 04:16 PM »
The only way being logged in, (or not), to the site would matter is if you are also doing searches or browsing at the same time, since you're increasing the number of requests per second.

It looks like they impose a rate limit of one request per second from the same IP, (strange I never hit it), unless it's from specific software, (eg. picard), or uses a recognisable UserAgent, (ie. a browser, PowerShell probably doesn't by default - I might look at how you can add it, if it's possible).

You can try putting in a Start-Sleep -seconds 1 after the for statement, that will delay the requests to about one per second.

Adjust the number up as necessary to increase the delay between requests, especially if you are browsing the site at the same time.

Or use Opera to browse the site with its internal VPN turned on so it looks like a different IP.

EDIT: Looks like PowerShell has its own UserAgent but can use one of the more standard browser settings which may decrease the rate limiting.  I'll look at it later today if I have time, or you can have a play ;)

https://stackoverflo...a-powershell-request
« Last Edit: November 08, 2020, 04:37 PM by 4wd »

fred dred

  • Supporting Member
  • Joined in 2010
  • **
  • default avatar
  • Posts: 12
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #19 on: November 09, 2020, 12:59 AM »
I was logged into the MB site, but I wasn't using it - I just added that info because 'more info is more better' when it comes to figuring out why it aint working :)

Adding a delay of 0.55 seconds to the loop seems to have fixed it  :Thmbsup:

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,538
    • View Profile
    • Donate to Member
Re: IDEA - Create a text doc of an artists discography.
« Reply #20 on: November 09, 2020, 02:39 AM »
Now uses Chrome's UserAgent and can set a delay from the command line:

Code: PowerShell [Select]
  1. <#
  2.   .\DiscoFile-MB3.ps1 -uri <Uri> [-delay <seconds>]
  3.  
  4.   Given the URI from https://musicbrainz.org for an artist it will retrieve the
  5.   discography.
  6.  
  7.   Where: Uri     = MusicBrainz artist page URI
  8.          seconds = [optional] number of seconds delay between Web Requests (0.1-90)
  9.  
  10.  
  11.   eg.
  12.  
  13.   .\DiscoFile-MB3.ps1 -uri https://musicbrainz.org/artist/66c662b6-6e2f-4930-8610-912e24c63ed1
  14.  
  15.   .\DiscoFile-MB3.ps1 -uri https://musicbrainz.org/artist/66c662b6-6e2f-4930-8610-912e24c63ed1 -delay 10
  16.  
  17.   Will result in a file called 'AC_DC-MB - Discography.txt' in the same directory as
  18.   DiscoFile-MB3.ps1
  19.  
  20. #>
  21. param (
  22.         [Parameter(Mandatory=$true)][string]$Uri,
  23.         [ValidateRange(0.1,90)][decimal]$delay
  24. )
  25. $sleep = $false
  26. if ($delay -gt 0) {
  27.   $sleep = $true
  28. }
  29. $userAgent = [Microsoft.PowerShell.Commands.PSUserAgent]::Chrome
  30.  
  31. $web = (Invoke-WebRequest -Uri $Uri -UserAgent $userAgent).RawContent
  32.  
  33. $temp = $web -replace '[\s\S]+application\/ld\+json..([\s\S]+?)<\/script>[\s\S]*$', '$1'
  34. $json = $temp | ConvertFrom-Json
  35.  
  36. $outFile = ($json.name + "-MB - Discography.txt").Split([IO.Path]::GetInvalidFileNameChars()) -join '_'
  37. if (Test-Path $outFile) {
  38.   Remove-Item $outFile -Force
  39. }
  40.  
  41. for ($i = 0; $i -lt $json.album.Count; $i++) {
  42.   if ($sleep) {
  43.     Start-Sleep -seconds $delay
  44.   }
  45.   $temp = $json.album[$i].albumReleaseType
  46.   if ($temp -match 'AlbumRelease') {
  47.     $temp = $json.album[$i].albumProductionType
  48.     if (($temp -match '(LiveAlbum)|(StudioAlbum)') -and !($temp -match '(CompilationAlbum)|(SoundtrackAlbum)')) {
  49.       $albumUri = ($json.album[$i].'@id' + "?inc=releases&fmt=json") -replace '\.org\/', '.org/ws/2/'
  50.       $album = (Invoke-WebRequest -Uri "$($albumUri)" -UserAgent $userAgent) | ConvertFrom-Json
  51.       "$($album.releases[0].date.Substring(0,4)) - $($album.title)" | Out-File -FilePath $outFile -Append
  52.     }
  53.   }
  54. }
  55.  
  56. Get-Content $outFile | Sort-Object | Set-Content $outFile