A small tool to view real-world ActivityPub objects as JSON! Enter a URL
or username from Mastodon or a similar service below, and we'll send a
request with
the right
Accept
header
to the server to view the underlying object.
{
"@context": [
"https://www.w3.org/ns/activitystreams",
{
"ostatus": "http://ostatus.org#",
"atomUri": "ostatus:atomUri",
"inReplyToAtomUri": "ostatus:inReplyToAtomUri",
"conversation": "ostatus:conversation",
"sensitive": "as:sensitive",
"toot": "http://joinmastodon.org/ns#",
"votersCount": "toot:votersCount",
"Hashtag": "as:Hashtag"
}
],
"id": "https://rukii.net/users/tero/statuses/113305644851767063",
"type": "Note",
"summary": null,
"inReplyTo": null,
"published": "2024-10-14T11:55:34Z",
"url": "https://rukii.net/@tero/113305644851767063",
"attributedTo": "https://rukii.net/users/tero",
"to": [
"https://www.w3.org/ns/activitystreams#Public"
],
"cc": [
"https://rukii.net/users/tero/followers"
],
"sensitive": false,
"atomUri": "https://rukii.net/users/tero/statuses/113305644851767063",
"inReplyToAtomUri": null,
"conversation": "tag:rukii.net,2024-10-14:objectId=318395725:objectType=Conversation",
"content": "<p>There are many reasons for LLM/LMM "hallucinations" or creativity. An interesting one relates to synthetic data.</p><p>A common way to create synthetic data which mixes together images and text is to use image captions. For example, we could have an image of a cat, and a caption "cat looking happy". Then we can synthetically create multi-modal instruction following data from this by asking an <a href=\"https://rukii.net/tags/LLM\" class=\"mention hashtag\" rel=\"tag\">#<span>LLM</span></a> to create questions and answers about the caption, and take those question-answers by associating them with the image instead.</p><p>So we'll get training data for a large multi-modal model (LMM) with e.g questions: "Q: <image>What animal is in this picture? A: Cat", "Q: What is the mood of the cat? <image> A: Happy"</p><p>Ok, to "hallucinations": What if the caption says "Spot the cat is happy because he has been told he is a good cat."?</p><p>The questions about the photo become to be about information not necessarily present in the photo: "What is the name of the cat?", "Why is the cat happy?"</p><p>When an LMM is trained with such data, and then someone asks it to tell them about a picture of a cat, the model will happily "hallucinate", that the name of the cat is Spot, and all sorts of other details not visible in the photo.</p><p>This can of course be mitigated by refining the training data better with LLMs, and telling them to omit facts in captions unlikely to be visible in the photos. Or better yet, use LMMs we already have trained with less than optimal data to do the same kind of refinement, because even if they would be keen on telling more about a cat than meets the eye, they would also have at least some level of sense of what is actually visible in the photo, e.g. does the cat have a name tag or something.</p>",
"contentMap": {
"en": "<p>There are many reasons for LLM/LMM "hallucinations" or creativity. An interesting one relates to synthetic data.</p><p>A common way to create synthetic data which mixes together images and text is to use image captions. For example, we could have an image of a cat, and a caption "cat looking happy". Then we can synthetically create multi-modal instruction following data from this by asking an <a href=\"https://rukii.net/tags/LLM\" class=\"mention hashtag\" rel=\"tag\">#<span>LLM</span></a> to create questions and answers about the caption, and take those question-answers by associating them with the image instead.</p><p>So we'll get training data for a large multi-modal model (LMM) with e.g questions: "Q: <image>What animal is in this picture? A: Cat", "Q: What is the mood of the cat? <image> A: Happy"</p><p>Ok, to "hallucinations": What if the caption says "Spot the cat is happy because he has been told he is a good cat."?</p><p>The questions about the photo become to be about information not necessarily present in the photo: "What is the name of the cat?", "Why is the cat happy?"</p><p>When an LMM is trained with such data, and then someone asks it to tell them about a picture of a cat, the model will happily "hallucinate", that the name of the cat is Spot, and all sorts of other details not visible in the photo.</p><p>This can of course be mitigated by refining the training data better with LLMs, and telling them to omit facts in captions unlikely to be visible in the photos. Or better yet, use LMMs we already have trained with less than optimal data to do the same kind of refinement, because even if they would be keen on telling more about a cat than meets the eye, they would also have at least some level of sense of what is actually visible in the photo, e.g. does the cat have a name tag or something.</p>"
},
"updated": "2024-10-14T16:32:47Z",
"attachment": [],
"tag": [
{
"type": "Hashtag",
"href": "https://rukii.net/tags/llm",
"name": "#llm"
}
],
"replies": {
"id": "https://rukii.net/users/tero/statuses/113305644851767063/replies",
"type": "Collection",
"first": {
"type": "CollectionPage",
"next": "https://rukii.net/users/tero/statuses/113305644851767063/replies?only_other_accounts=true&page=true",
"partOf": "https://rukii.net/users/tero/statuses/113305644851767063/replies",
"items": []
}
}
}