• pebbles@sh.itjust.works
    cake
    link
    fedilink
    English
    arrow-up
    1
    ·
    5 hours ago

    I would reccomend you look into sythetic data. It is a real thing and is used intentionally in basically every LLM of the past year or more. Last I heard of an LLM without sythetic data was Dots LLM1 and that wasn’t competitive when it came out.

    • Jiral@lemmy.world
      link
      fedilink
      arrow-up
      1
      ·
      2 hours ago

      I did read up a bit on synthetic data. Sounds like it is mainly used to dilute human input, or rather multiply it artificially. It does appear it is tricky and can have downsides (especially when you remove the human content too much or entirely, but that of course also depends on the precise field of training). In protein engineering there is the saying, you get what you screen for, and that is primarily used as word of caution. I would assume the same is true for models, you get what you train for.