I’m a retired Unix sysadmin. Over the years I’ve built things in COBOL, FORTAN, C, perl, rexx, PHP, visual basic, various Unix shells and maybe others. Nothing has been a real “application” - mostly just utilities to help me get things done.

Now that I’m retired, and it’s cold outside, I’m curious to try some more coding - and I have an idea.

The music communities here seem to post links to YouTube. I generally use Lemmy on my phone but don’t use YouTube, or listen to music, on my phone if I can help it. I’d like to scrape a music community here and add the songs posted to a playlist in my musicbrainz account.

Does that sound like a reasonable learner project? Any suggestions for language and libraries appreciated. My preferred IDE is vim on bash and I have a home server running Linux where this could run as a daemon, or be scheduled.

  • Diabolo96@lemmy.dbzer0.com
    link
    fedilink
    arrow-up
    12
    arrow-down
    3
    ·
    11 months ago

    The best programming language for automating things is python. Python is easy and comes with a lot of modules that allow you to do anything and everything, I guarantee you that once you start automating stuff it’ll become like a drug and you’ll just “automate it” whenever you have anything repetitive.

    And BTW, one of the main uses of python is website scraping.

    https://musicbrainz.org/doc/MusicBrainz_API

    • some_guy@lemmy.sdf.org
      link
      fedilink
      arrow-up
      13
      arrow-down
      1
      ·
      11 months ago

      The best language for automation is the one you know best. The second best is one you have to learn.

      I think you could do this in bash with YouTube-dl.

      • Diabolo96@lemmy.dbzer0.com
        link
        fedilink
        arrow-up
        1
        ·
        edit-2
        11 months ago

        Indeed. while my bash-fu is redimentary at best, I don’t think Bash can be used for web scrapping ? But I think he could use RSS to get the posts, then extract youtube links with Regex and use the dump feature of yt-dlp* to get the video category, title,etc by using jq to parse the json. Then, it’s probably just a matter of using curl to do the API calls and voilà.

        *yt-dlp is better maintained than youtube-dl, or so I heard.

        • some_guy@lemmy.sdf.org
          link
          fedilink
          arrow-up
          3
          arrow-down
          1
          ·
          10 months ago

          I built two scrapers for a website that hosts images and videos using bash.

          They’re educational, I swear! /s

          I looked through the html and figured out regexes for their media. The scripts will parse all the links on the thumbnail pages and then load the corresponding primary pages with curl. On those pages, it then uses wget to grab the file. Some additional pattern matching names the file to the name of the post.

          It’s probably convoluted, but you can accomplish a lot in bash if you want to.

    • Great Blue Heron@lemmy.caOP
      link
      fedilink
      arrow-up
      7
      ·
      11 months ago

      I find Python difficult - no idea why, it just doesn’t feel right. I’ve tried a few times but never been able to do anything useful with it - that’s why it’s not in my list above. It does seem though that my proposed project, and development “style”, is best suited to Python. Maybe it’s time to try again.

      • Diabolo96@lemmy.dbzer0.com
        link
        fedilink
        arrow-up
        3
        ·
        edit-2
        11 months ago

        It was just a recommendation. If you feel like python isn’t for you, you can try any other language and the only difference will be how much time it’ll take to make it, but otherwise you can use C if you want. Maybe you’re so used to low level programming like managing memory and having to declare types everywhere that python dumb proof approach is difficult for you. Just don’t think too hard about it, if it’s a personal use script then there’s no need to think about it’s efficiency or ugliness. If it works, it works.

      • some_guy@lemmy.sdf.org
        link
        fedilink
        arrow-up
        5
        arrow-down
        2
        ·
        11 months ago

        If you work in bash and don’t like python, maybe it’s too strict. Look into Ruby. It was inspired by Perl. I found it more to my style in that there are many correct solutions and not one implied correct solution.

      • onlinepersona@programming.dev
        link
        fedilink
        English
        arrow-up
        3
        arrow-down
        2
        ·
        edit-2
        10 months ago

        Python is basically runnable pseudo code that you would write on a napkin to explain stuff to somebody. There you don’t care about curly backets and naturally indent to show scope. It’s way simpler C and if you want to, you can add type hints (aka faux static typing).

        Package management is done with pip although nowadays poetry is better as it uses one file to define everything about your project and configure the tools (linter, tester, autoformatter, static type checking)

        The advantage of python is that it has lots and lots of libraries. You don’t need to fiddle around with the lemmy API - use a library:

        Want to connect to musicbrainz? https://pypi.org/project/musicbrainzngs/ is probably the best.

        –>

        Create a virtual env (basically allows you to install all your project dependencies in an environment separate from the global one): python3 -m venv .venv.

        Activate the virtual in your shell source .venv/bin/activate.

        Now you can start installing dependencies. If you want it super simple, use pip install $package, but updating the list of packages you want in your project is manual: pip freeze > requirements.txt (install them again with pip install -r requirements.txt after rm -rf .venv should you want to start fresh) and you can run into problems with clashing dependencies.
        So, I recommend using poetry pip install poetry. poetry new . to setup basic project structure, then add runtime dependencies with poetry add $package e.g poetry add pylemmy musicbrainzngs.

        It’s possible to add dev dependencies with poetry like ruff for linting and autoformatting your code and mypy for static type checking. Your unit tests can be written using unittest from the standard library.

        CC BY-NC-SA 4.0

        • Great Blue Heron@lemmy.caOP
          link
          fedilink
          arrow-up
          1
          ·
          10 months ago

          Thank you for your detailed response. It’s a bit much for my proposed “project”. I won’t be using any libraries (other than built-in python json etc.). I’ve prototyped most of it and it’s currently about 15 lines of code. Literally one call to lemmy, a search to Musicbrainz and a playlist update to listenbrainz. I know it will grow lots as I make it a bit more robust, but it’s still very small.

      • Outcide@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        10 months ago

        Another (mostly) retired Unix sysadmin here. I never could make Python work in my brain, but last year discovered Svelte/SvelteKit and really like it. I’d always kinda hated on JS, but actually it’s pretty nice these days.

  • ericjmorey@programming.dev
    link
    fedilink
    arrow-up
    7
    ·
    edit-2
    10 months ago

    I’m going to assume you’re going to go ahead with the idea. You could make use of the fact that every lemmy based community has an RSS feed.

    • Great Blue Heron@lemmy.caOP
      link
      fedilink
      arrow-up
      4
      ·
      11 months ago

      Yes, I’m working on it now. Struggling with basic stuff like pulling values out of the json returned by the API when I ask for a list of posts. Python really does not click for me, but I’m determined, for now, to keep at it. An the RSS feed seems like a much easier (than what…?) way to just get new posts with each run - thank you!

  • Aurenkin@sh.itjust.works
    link
    fedilink
    arrow-up
    5
    ·
    11 months ago

    Sounds like you have some programming experience already but whether this is a reasonable learner project I think depends a bit in what kind of things you’re trying to learn but it sounds pretty good to me.

    If you’re going to be using a familiar language and you already know how to call web APIs pretty comfortably I think that’s a pretty reasonable pet project assuming musicbrainz has a public API (I’m not familiar with it unfortunately).

    If you’re wanting to learn some more basic stuff around calling APIs and maybe a language you’re not familiar with I would start by making something super dumb that calls an unauthenticated public API like a weather service or something.

    Ultimately I think your project idea sounds good though, just making a suggestion for something even simpler you could try as a stepping stone if you need to.

    Either way congrats on your retirement! Have fun with all the interesting projects you decide to pursue.

  • hallettj@beehaw.org
    link
    fedilink
    English
    arrow-up
    3
    ·
    11 months ago

    That sounds like a good learning project to me. I think there are two approaches you might take: web scraping, or an API client.

    My guess is that web scraping might be easier for getting started because scrapers are easy to set up, and you can find very good documentation. In that case I think Perl is a reasonable choice of language since you’re familiar with it, and I believe it has good scraping libraries. Personally I would go with Typescript since I’m familiar with it, it’s not hard (relatively speaking) to get started with, and I find static type checking helpful for guiding one to a correctly working program.

    OTOH if you opt to make a Lemmy API client I think the best language choices are Typescript or Rust because that’s what Lemmy is written in. So you can import the existing API client code. Much as I love Rust, it has a steeper learning curve so I would suggest going with Typescript. The main difficulty with this option is that you might not find much documentation on how to write a custom Lemmy client.

    Whatever you choose I find it very helpful to set up LSP integration in vim for whatever language you use, especially if you’re using a statically type-checked language. I’ll be a snob for just a second and say that now that programming support has generally moved to the portable LSP model the difference between vim+LSP and an IDE is that the IDE has a worse editor and a worse integrated terminal.

  • nickwitha_k (he/him)@lemmy.sdf.org
    link
    fedilink
    arrow-up
    1
    ·
    10 months ago

    If Python isn’t clicking with you and you want to try a newer language with a good standard library, maybe try Go. It is based on C but, is a bit easier to iterate in development. Plus, it was designed with network applications in mind so, it handles JSON pretty well. As an added bonus, it is compiled, making (from my perspective) running a daemon a bit more straightforward.

    As an aside, I am currently a software engineer and use (Neo)vim as my IDE, because of the comfort that I built up with the terminal and vim as a sysadmin. It works great as an IDE, especially with LSP and other QoL plugins.

    • Great Blue Heron@lemmy.caOP
      link
      fedilink
      arrow-up
      1
      ·
      10 months ago

      I’m a little determined to stick with Python because I feel that I should - everyone should be able to code Python :-)

      The main problem I have with it is the complex, relaxed, data structures. I’m finding that the type() command in interactive mode is helping a lot. I’m having lots of moments like - “Ah, I’m not down to the dict yet, I’m still in the list…”

      • nickwitha_k (he/him)@lemmy.sdf.org
        link
        fedilink
        arrow-up
        1
        ·
        10 months ago

        The way that I see it is that some languages will “click” while others may not. Nothing wrong with that. I like your determination!

        I’d say, from your stated problem, that you may be selling yourself short a bit. My colleagues and I frequently run into the same when debugging. And the approach that you are using to work through it is exactly what I’d recommend as well (along with type hinting). I also find the Python interactive shell great for testing concepts and do one-off things that are easier in Python than Bash.