I shared a link from NPR Tiny Desk but there was no image preview or video embed. After investigating this I learned that when you do that, each instance picks up the link and individually retrieves the link preview nearly simultaneously. For gargantuan sites like YT or Reddit, this is a trivial amount of traffic and not a problem, but for smaller sites like NPR this results in a sort of botnet DDOS attack on the site’s servers. More here. Another example here . For that reason, NPR has blocked Fediverse servers like Mastodon.

I have experienced this on my personal server as well (DDOS), but at the time didn’t know why, exactly.

This isn’t the end of the world, but certainly is not great and hurts engagement with the content. Is there a solution for this? Perhaps using some sort of webtorrent service like PeerTube where they just fetch the information from another fediverse server instead of the source server? Maybe fetching embeds client-side? At least that would stagger the requests.

  • FundMECFS@piefed.zip
    link
    fedilink
    English
    arrow-up
    53
    ·
    10 hours ago

    Thanks for bringing this up. I think in general we need to work on optimisation of Activitypub and Threadiverse implementations of it.

  • GenosseFlosse@feddit.org
    link
    fedilink
    English
    arrow-up
    37
    ·
    10 hours ago

    Is there a reason why the publishing instance is not downloading the preview image, and then submits it with the post to the other instances? This way they would not have to fetch it themselves.

    • Funkt4st1c@lemmy.world
      link
      fedilink
      English
      arrow-up
      29
      ·
      10 hours ago

      Yeah, thats how it should be: one download, everyone else downloads from that server, not the original source.

      Seems abusable though

      • _haha_oh_wow_@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        9
        ·
        5 hours ago

        It is, but that’s what defederation is for: If there are instances that are being sketchy, they are supposed to get cut off from the rest of the Fediverse.

      • Gibibit@lemmy.world
        link
        fedilink
        English
        arrow-up
        18
        arrow-down
        2
        ·
        9 hours ago

        Seems abusable though

        If one instance doesn’t trust another, they shouldn’t federate in the first place.

      • a4ng3l@lemmy.world
        link
        fedilink
        English
        arrow-up
        3
        ·
        9 hours ago

        Add metadata with a checksum of the og one with signature and let each client check that up.

        • inzen@lemmy.world
          link
          fedilink
          English
          arrow-up
          4
          ·
          7 hours ago

          If it’s a signed checksum then maybe. But if the og instance is doing the abusing then it won’t help.

          • a4ng3l@lemmy.world
            link
            fedilink
            English
            arrow-up
            1
            ·
            7 hours ago

            Yeah there must be some trust indeed of the og instance. A scoring system could help with that.

            • prenatal_confusion@feddit.org
              link
              fedilink
              English
              arrow-up
              1
              ·
              4 hours ago

              Or … Or we could stagger the requests of the different instances so that they don’t overload the linked site. Why reinvent the wheel?

              • a4ng3l@lemmy.world
                link
                fedilink
                English
                arrow-up
                1
                ·
                4 hours ago

                Eventually that leaves the load to the source event if it’s differed. And the load would still grow along the growth of the fediverse. So this would likely be pushing the issue to tomorrow. Signing content checksum isn’t exactly like reinventing a wheel, it’s something done in many contexts. And it removes the load from the source entirely.

  • cybervegan@lemmy.world
    link
    fedilink
    English
    arrow-up
    30
    ·
    11 hours ago

    I have my blog address (which goes to my home server) in my Mastodon profile, and every time I post on Masto, my server gets a request from every Mastodon site that federates with my instance - i.e. hundreds of requests. It’s no problem for my server, because my blog is a static site, but for anything that dynamically generates every single page impression (like worpress or joomla), it could be a problem. Caching settings might help, I suppose.

    Truth is, though, I get far more traffic from vulnerability scanners and AI crawlers that don’t honour robots.txt (or aren’t yet listed).

    • 0x0@lemmy.zip
      link
      fedilink
      English
      arrow-up
      9
      ·
      6 hours ago

      that don’t honour robots.txt (or aren’t yet listed).

      Throw’em in the tar pit.

  • dropdrip@lemmy.ml
    link
    fedilink
    English
    arrow-up
    34
    arrow-down
    3
    ·
    12 hours ago

    It’s almost like the software that powers the internet was designed for a different time and is only still used because there’s monied interest in keeping it around. We need GNUnet.

    • milbyte@lemmy.ml
      link
      fedilink
      English
      arrow-up
      18
      arrow-down
      1
      ·
      10 hours ago

      is there something that makes this different to 17 other proposals for a new internet or is it just a case of https://xkcd.com/927/

    • sem@lemmy.blahaj.zone
      link
      fedilink
      English
      arrow-up
      12
      ·
      11 hours ago

      GNUnet helps building a new Internet

      GNUnet is a network protocol stack for building secure, distributed, and privacy-preserving applications.

      With strong roots in academic research, our goal is to replace the old insecure Internet protocol stack.

      GNUnet provides privacy-by design

      It provides [improving addressing, routing, naming and content distribution](https://www.w3.org/2014/strint/papers/65.pdf) in a technically robust manner - as opposed to ad-hoc designs in place today.

  • ByteJunk@lemmy.world
    link
    fedilink
    English
    arrow-up
    45
    ·
    15 hours ago

    I mean, just how many instances are gonna ping the URL? Several hundred, low thousands at worst?

    Unless it happens to be some very large media content or a dynamic page with no caching, I’d think the automated federation hits shouldn’t be too bad.

    The influx of users that might come after a viral post though, that’s rough…

    • Doom@lemmy.world
      link
      fedilink
      English
      arrow-up
      8
      ·
      8 hours ago

      Yeah but we can’t say we want to encourage people to have/build/share independent smaller websites then beat them up when they are linked. That’s a huge mismatch of morality. Either we want to encourage people off the big platforms and should make it easier for the little guys or we are blowing magic smoke.

    • dreamkeeper@lemmy.world
      link
      fedilink
      English
      arrow-up
      19
      arrow-down
      1
      ·
      12 hours ago

      It’s just bad design. What happens when/if the fediverse scales up to many more instances?

      • 1984@lemmy.today
        link
        fedilink
        English
        arrow-up
        5
        arrow-down
        1
        ·
        10 hours ago

        I liked writing efficient code but every place I worked didn’t care at all about it. All they wanted was a finished feature and a kanban note moved to finished.

  • wjs018@piefed.wjs018.xyz
    link
    fedilink
    English
    arrow-up
    192
    ·
    20 hours ago

    There are two proposed solutions I have seen. I do not claim that this is exhaustive, just only limited to the proposals that I have stumbled across.

    FASP - Fediverse Auxiliary Service Provider

    The idea that mastodon has is to have a third-party service that does something like generate a link preview once and then share that to fediverse instances that need it. They have a whole spec they wrote up about these kinds of services that can be run to help out fediverse instances with different things like this, search, or discoverability. They call these FASPs - the repo with the spec is here.

    I don’t yet know of any FASP that is live and running, so it is kind of a paper spec at the moment. I don’t do too much on the microblogging side of the fediverse though, so I very well could be wrong.

    Federate link metadata and the post at the same time

    This comes from an issue on PieFed’s repo. It is basically just an idea at this point, but it is a pretty basic one; have the originating instance fetch the link preview information and simply federate it out alongside the post itself.

    • jaybone@lemmy.zip
      link
      fedilink
      English
      arrow-up
      118
      ·
      20 hours ago

      That second solution is what I thought of as doing as I read this. It doesn’t rely on a third party service being up. And that instance has to do that work anyway. So why not just share the results with others?

      • khannie@lemmy.world
        link
        fedilink
        English
        arrow-up
        9
        ·
        16 hours ago

        It was my first thought too but there are downsides to it. Trust is a small one given the nature of the system but compute and bandwidth costs could be an issue also, especially for larger instances.

        The first solution is more fragile but could be CDN’d and those are cheap.

        • Kangae_Hishiryo@scribe.disroot.org
          link
          fedilink
          English
          arrow-up
          4
          ·
          6 hours ago

          WebTorrent kind of solution like PeerTube is the best, because it’s highly scalable yet not that resource-intensive, because everyone (even users) provides their bit; also, as it’s entirely P2P, it acts sort like a distributed and decentralized CDN, so it’s snappier for everyone because it’s more likely that you have a peer near to you like that.

          • khannie@lemmy.world
            link
            fedilink
            English
            arrow-up
            4
            ·
            4 hours ago

            Absolutely LOVE this idea but I feel like a solid implementation would be a pain in the fanny (and it also requires other peers to be online). I dip in and out over the day myself but I don’t know what others use case is.

            Also it is far less trustworthy and harder to block bad actors (V’s removing a known bad actor from a relatively small peer group).

            • Kangae_Hishiryo@scribe.disroot.org
              link
              fedilink
              English
              arrow-up
              2
              ·
              4 hours ago

              Idk, the part of uptime of peers is solved precisely by the fact that not every peer will be connected at the same moments that every other peer. If I’m connected at 10pm, it’s highly likely that there will be at least another single peer at that hour that has what I’m pulling.

              As for bad actors, that’s something that at the end of the day is unavoidable. Though with some automated systems could be possible to make it more hard for them, but it’s not that different from a centralised approach, and thus is better because is computationally more efficient.

        • Ghoelian@piefed.social
          link
          fedilink
          English
          arrow-up
          6
          ·
          13 hours ago

          I don’t think compute resources should be an issue, if anything it saves other instances resources.

          As far as I understand, right now, all instances are fetching this data individually, so they all have the same compute load from this. Doing it on one and sending it along with federation should keep resource useage pretty much the same on the originating instance, and less for all others.

          This would still increase bandwidth of course, but at least with all VPS providers I have used, bandwith is pretty cheap.

        • 4am@lemmy.zip
          link
          fedilink
          English
          arrow-up
          110
          ·
          20 hours ago

          The same way you know the rest of the data hasn’t been tampered with - you don’t. And when you find untrustworthy instances, you defederate from them.

          • SorteKanin@feddit.dk
            link
            fedilink
            English
            arrow-up
            15
            arrow-down
            1
            ·
            13 hours ago

            That’s not true, you know the data hasn’t been tampered with because of HTTP signatures. The fediverse isn’t just “yolo, just trust me bro”. There are some parts that are a bit more towards that side (particularly with groups on Lemmy) but I don’t think this should be.

            • Natanael@infosec.pub
              link
              fedilink
              English
              arrow-up
              6
              ·
              7 hours ago

              HTTP signatures are not a default feature, the baseline is ECDHE key exchange with forward secrecy and deniability. Headers are not typically signed.

              There isn’t even an official standard for redistributable signed page metadata. There’s just prototypes. I absolutely hope one of those prototypes becomes official, but we’re not there yet.

              The activitypub fediverse is literally very extremely Yolo trust me bro, it works because you can see the originating server with everything and misbehaving ones gets banned.

        • jaybone@lemmy.zip
          link
          fedilink
          English
          arrow-up
          43
          ·
          20 hours ago

          I’m trusting the user who submitted the link and/or the instance that user is on. I’m doing that anyway by clicking on their link right?

        • rako@tarte.nuage-libre.fr
          link
          fedilink
          Français
          arrow-up
          4
          ·
          15 hours ago

          So I receive a post from someone presumably with some comment, along with a preview of a link, and I’m supposed to trust one but not the other ?

    • tuckerm@feddit.online
      link
      fedilink
      English
      arrow-up
      68
      ·
      19 hours ago

      I’m a fan of the second one. My thought is, if you trust a server enough to accept messages from it, then you should also trust it enough to let it scrape the page title, article image, etc. for you, too.

      • shrek_is_love@lemmy.ml
        link
        fedilink
        English
        arrow-up
        15
        ·
        19 hours ago

        Seems like a no-brainer! They could even leave the old behavior as an option for instance admins to re-enable, but if they default it to not DDoSing, that would make a huge difference.

      • timochka@lemmy.zip
        link
        fedilink
        English
        arrow-up
        2
        arrow-down
        1
        ·
        12 hours ago

        The great thing about this if you run a server, as soon as a user posts a link to some kiddy porn, you not only download it automatically, you distribute it as well. Go-go-gadget-jailtime.

        (Actually already a problem with Mastodon federation of image posts, and why I would strongly encourage against anyone in a strict liability jurisdiction from running their own Mastodon server tbh.)

        • tuckerm@feddit.online
          link
          fedilink
          English
          arrow-up
          2
          ·
          4 hours ago

          This is no different from how it is already, though. If a user posts a CSAM link, your server immediately visits that link to scrape it, and you suddenly have that content on your server.

          Of course, if you have a blocklist of forbidden sites, then your server will know not to scrape that site. And the same thing applies if we allow servers to send pre-scraped metadata. If the incoming message is for a blocked URL, just drop the message and don’t save any part of it.

          The only new threat here is for a malicious server operator to modify outgoing messages to include CSAM metadata on a harmless link. However, that possibility already exists with any other content posted on the fediverse. If a malicious server is modifying its own outgoing messages, the only fix is to defederate from it.

          Allowing servers to forward pre-scraped metadata to other servers does not introduce any new threats that do not already exist.

        • Wildmimic@anarchist.nexus
          link
          fedilink
          English
          arrow-up
          5
          ·
          10 hours ago

          There’s no reason to believe that this mechanism doesn’t check for CSAM (I am also pretty stumped that this is actually a problem with Mastodon according to your comment?) . Lemmy had a phase where messed up stuff got federated, which led to @db0@lemmy.dbzer0.com’s implementation of an anti-CSAM tool: https://github.com/db0/fedi-safety, which is based on the CSAM detection tool for the AI Horde: https://dbzer0.com/blog/ai-powered-anti-csam-filter-for-stable-diffusion/ - it’s probably not so hard to adapt for Mastodon too.

          • timochka@lemmy.zip
            link
            fedilink
            English
            arrow-up
            1
            arrow-down
            1
            ·
            10 hours ago

            If you want to trust a hypothetical “detects CSAM honest” feature with something like this, well, you go ahead.

            I mean; just imagine for one moment that lemmy.ml actually is the hotbed of revolutionaries that is going to usher in the Glorious Revolution that it thinks it is; it starts actually moving mountains, politically speaking.

            And, some bright spark has implemented the “automatically download and distribute content from unverified sources, protected only by the open-source Kiddy-Porno-Matic Verifier v0.1.23” feature.

            If I were Peter Thiel it would take me 5 minutes to start posting links that caused lemmy.ml’s servers to start downloading and then distributing kiddy porn, about 20 minutes to get some tame journalists parked outside the owners’ houses and lemmy.ml’s datacentre, and another 10 minutes to call up the friendly neighbourhood goons in wherever they are located for the photogenic armed raid. And once the “revolutionary leaders in court for making and distributing child pornography” headlines had finished writing themselves, I wouldn’t even worry about whether the court would accept the “but we ran the Kiddy-Porno-Matic Verifier v0.1.23!” defence, because their lives would be over already. No smoke without fire, after all…

            Nah. “Automatically download and distribute content” features need rather more thinking through than historically Mastodon et al have given them. They certainly don’t need to create any more.

            (And to be clear - maybe Mastodon has fixed some of their underlying problems - I ran an instance a few years ago, and the constant whack-a-mole of blocking actual-Nazi and actual-Pedo instances, and the risk that it could just go ahead and download something to my servers because a user followed the wrong person had me shutting it down not long after. As I say, maybe they fixed that - but this mooted feature would be bringing back the same problem.)

            Of course, maybe Lemmy.ml is (a) hosted in a jurisdiction where they don’t give a fuck, and (b) can afford to defend themselves. But this kind of feature will actively work against people who otherwise might host a small instance, and I thought that is what we wanted, not just a few massive instances that can afford to lawyer up.

            All, I should say, IMHO of course.

        • Pamasich@kbin.earth
          link
          fedilink
          arrow-up
          1
          ·
          1 hour ago

          Sorry, I think I’m just blind. I tried a non piefed.social post earlier today and didn’t see it in the JSON, but checking a piefed.social post now, the property is there. So either I just didn’t see it the first time around, or the instance I tried it with wasn’t the best example to use.

          Anyway, deleted my comment. But then OP’s post confuses me again, because there really shouldn’t be a DDoS happening toward the small site. The instance hosting the thumbnail sure, but not the site that’s being linked to.

    • panos@catodon.rocks
      link
      fedilink
      English
      arrow-up
      1
      ·
      10 hours ago

      But what happens when a link target changes? For example, I’ve seen news articles change as the author gets new information. Might be an edge case, but a solution like this would always show the preview at the time of the initial post.

      • HubertManne@piefed.social
        link
        fedilink
        English
        arrow-up
        1
        ·
        6 hours ago

        is this a bad thing though. like if im the poster I don’t necessarily want the content to change compared to what I saw and was excited about so came to tell people about.

    • Rimu@piefed.social
      link
      fedilink
      English
      arrow-up
      2
      ·
      14 hours ago

      Really the only metadata we are interested in is the thumbnail image url. Afaik PieFed already retrieves this from the og:image meta tag before sending the URL along with the other post content to the instance hosting the community.

      Another idea - those meta tags are also retrieved during post composition in an ajax call, in order to prepopulate the title and body fields so the image url could be stashed in a hidden field at that point…

  • Rimu@piefed.social
    link
    fedilink
    English
    arrow-up
    75
    ·
    20 hours ago

    Lemmy does this to itself, too - when you create a post it federates out and then every Lemmy instance that receives it makes a GET request to retrieve the post from the original server. Even though it just received the exact same data. Some other fedi software is even worse - Akkoma does a GET of the id of every activity it receives, not just for posts but for Likes too.

    It’s insane.

    Anyway, proper caching at the reverse proxy / Cloudflare level solves it. The same kind of caching setup would do the same for sites serving HTML, like NPR too.

    • Ghoelian@piefed.social
      link
      fedilink
      English
      arrow-up
      5
      ·
      13 hours ago

      That’s weird. Is that how it’s supposed to work, or did they just implement it in a bad way?

      • Rimu@piefed.social
        link
        fedilink
        English
        arrow-up
        2
        ·
        3 hours ago

        I’m not sure but my guess it’s an attempt to deal with the problem where sometimes federation activities arrive in the wrong order (especially after an instance has been offline for a while) - by making a GET to the origin server you can be sure you’ve got the latest version of the content. There are probably better ways to achieve that, though.

        PieFed prioritises efficiency over correctness so it just accepts whatever is sent to it. In 3 years no one has complained about seeing stale content although I’m sure it happens occasionally.

      • The_Decryptor@aussie.zone
        link
        fedilink
        English
        arrow-up
        10
        ·
        11 hours ago

        It’s an access control method, called “authorized fetch”. Rather than broadcasting the data you just announce the source of it, and then any client that wants to show it needs to negotiate with the originating server to request the data (And then blocklists and such come into play)

        Say you’ve got 3 users, A, B, and C. A has blocked C, B follows A, and C follows B. Now A makes a post, B sees it in their feed and reboosts it, what does C see?

        If B’s server sends the full response along, then C can see the contents of A’s post even though they’ve been blocked by A. But if all it sends is the URL of the post, C’s server needs to make a request for it (providing authentication), and then A’s server can reject it, so C sees nothing.

    • SorteKanin@feddit.dk
      link
      fedilink
      English
      arrow-up
      1
      ·
      13 hours ago

      How else can you be sure the content is valid though? Lemmy posts are sent as Announces from the community, not from the actor posting the post. The community could just fake a post if you just trust it.

  • Pamasich@kbin.earth
    link
    fedilink
    arrow-up
    10
    ·
    15 hours ago

    I don’t really get it. The image is federated, there’s no need for instances to refetch it. Just checked out some posts and indeed it’s there, in the image property. With the URL pointing to the Lemmy instance.

    So if this is actually an issue, that’s just bad programming and needs to be fixed.

    Edit: didn’t notice OP uses Piefed. Unlike Lemmy and Mbin, it seems like Piefed doesn’t federate it.

    Edit: actually I’m blind, Piefed does federate it too, sorry piefed

      • Pamasich@kbin.earth
        link
        fedilink
        arrow-up
        8
        ·
        12 hours ago

        I just took this random post from my frontpage: https://sh.itjust.works/post/68605733 and made a manual request for the federated JSON-LD.

        {
          "@context": [
            "https://join-lemmy.org/context.json",
            "https://www.w3.org/ns/activitystreams"
          ],
          "type": "Page",
          "id": "https://sh.itjust.works/post/68605733",
          "attributedTo": "https://sh.itjust.works/u/derrickoswald",
          "to": [
            "https://lemmy.world/c/technology",
            "https://www.w3.org/ns/activitystreams#Public"
          ],
          "name": "Stratospheric airship flew from US to Japan and back",
          "cc": [],
          "content": "<p>It seems with an unmanned airship they should be using hydrogen for the ~8% increased payload capacity.</p>\n",
          "mediaType": "text/html",
          "source": {
            "content": "It seems with an unmanned airship they should be using hydrogen for the ~8% increased payload capacity.",
            "mediaType": "text/markdown"
          },
          "attachment": [
            {
              "href": "https://arstechnica.com/gadgets/2026/10/trans-pacific-journey-tested-stratospheric-cell-tower-concept-in-japan/",
              "mediaType": "text/html; charset=utf-8",
              "type": "Link"
            }
          ],
          "image": {
            "type": "Image",
            "url": "https://sh.itjust.works/pictrs/image/df608ab3-b644-489d-b217-9224aa09e203.jpeg"
          },
          "sensitive": false,
          "published": "2026-10-19T20:16:37.383385Z",
          "language": {
            "identifier": "en",
            "name": "English"
          },
          "audience": "https://lemmy.world/c/technology",
          "tag": [
            {
              "href": "https://sh.itjust.works/post/68605733",
              "name": "#technology",
              "type": "Hashtag"
            }
          ]
        }
        

        Notice the

        "image": {
            "type": "Image",
            "url": "https://sh.itjust.works/pictrs/image/df608ab3-b644-489d-b217-9224aa09e203.jpeg"
          }
        

        Yes the image itself isn’t sent over (iirc they never are?), but it’s stored on sh.itjust.works. Other instances grab it from there, not the website the post links to.

        I did notice just now though that some Lemmy instances seem to just proxy the image through their servers, which I assume is to avoid CORS. But that’s an instance choice, not a general fediverse issue.

        Mbin does the same as Lemmy, but it also includes the image in the attachment property. Probably for compatibility with some other platform.

        One thing I didn’t notice before was that you’re using Piefed. That’s the one software I didn’t check before writing my previous comment. It does seem like Piefed is the outlier here. I tried a post from piefed.ca and the image is nowhere to be seen. The image property is not present, it’s not in the attachment array, and I can’t see any other property that contains it.

        Edit: A piefed.social post I just tried did have an image, so I assume I’m just blind.

      • Ghoelian@piefed.social
        link
        fedilink
        English
        arrow-up
        3
        ·
        13 hours ago

        Are you sure? In your post you’re talking about fetching an image for a link preview, not an image post.

        Usually, when I see an image post, the source is actually the pictrs of the source instance, so in that case I think it is federated.

          • Ghoelian@piefed.social
            link
            fedilink
            English
            arrow-up
            4
            ·
            12 hours ago

            I figured a post with a link with preview image wouldn’t be the same in terms of event as an image post. Turns out it pretty much is, but also, I checked a few posts and cross-posts across instances, all of them have an image url of the original instance, not the one I’m viewing on. So if they are individually fetching this image, they’re not using it.

            Edit: I was wrong, two of the crossposts just didn’t respond to the request for activity, so I was looking at the wrong response data

          • Goodlucksil@lemmy.dbzer0.com
            link
            fedilink
            English
            arrow-up
            1
            ·
            10 hours ago

            The image isn’t federated, but [I believe] it should be.

            But I am already thinking it shouldn’t be because of hosting concerns.