finetune_gen.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"Anarchism is a political philosophy and movement that is sceptical of authority and rejects all involuntary, coercive forms of hierarchy. Anarchism calls for the abolition of the state, which it holds to be unnecessary, undesirable, and harmful. As a historically left-wing movement, placed on the farthest left of the political spectrum, it is usually described alongside communalism and libertarian Marxism as the libertarian wing (libertarian socialism) of the socialist movement, and has a strong historical association with anti-capitalism and socialism.\\n\\nHumans lived in societies without formal hierarchies long before the establishment of formal states, realms, or empires. With the rise of organised hierarchical bodies, scepticism toward authority also rose. Although traces of anarchist thought are found throughout history, modern anarchism emerged from the Enlightenment. During the latter half of the 19th and the first decades of the 20th century, the anarchist movement flourished in most parts of the world and had a significant role in workers' struggles for emancipation. Various anarchist schools of thought formed during this period. Anarchists have taken part in several revolutions, most notably in the Paris Commune, the Russian Civil War and the Spanish Civil War, whose end marked the end of the classical era of anarchism. In the last decades of the 20th and into the 21st century, the anarchist movement has been resurgent once more.\\n\\nAnarchism employs a diversity of tactics in order to meet its ideal ends which can be broadly separated into revolutionary and evolutionary tactics; there is significant overlap between the two, which are merely descriptive. Revolutionary tactics aim to bring down authority and state, having taken a violent turn in the past, while evolutionary tactics aim to prefigure what an anarchist society would be like. Anarchist thought, criticism, and praxis have played a part in diverse areas of human society. Criticism of anarchism include\", 'Q: What is the main goal of anarchism?\\nA: The main goal of anarchism is the abolition of the state.']\n",
      "[' claims that it is internally inconsistent, violent, or utopian.\\n\\nEtymology, terminology, and definition \\n\\nThe etymological origin of anarchism is from the Ancient Greek anarkhia, meaning \"without a ruler\", composed of the prefix an- (\"without\") and the word arkhos (\"leader\" or \"ruler\"). The suffix -ism denotes the ideological current that favours anarchy. Anarchism appears in English from 1642 as anarchisme and anarchy from 1539; early English usages emphasised a sense of disorder. Various factions within the French Revolution labelled their opponents as anarchists, although few such accused shared many views with later anarchists. Many revolutionaries of the 19th century such as William Godwin (1756–1836) and Wilhelm Weitling (1808–1871) would contribute to the anarchist doctrines of the next generation but did not use anarchist or anarchism in describing themselves or their beliefs.\\n\\nThe first political philosopher to call himself an anarchist () was Pierre-Joseph Proudhon (1809–1865), marking the formal birth of anarchism in the mid-19th century. Since the 1890s and beginning in France, libertarianism has often been used as a synonym for anarchism and its use as a synonym is still common outside the United States. Some usages of libertarianism refer to individualistic free-market philosophy only, and free-market anarchism in particular is termed libertarian anarchism.\\n\\nWhile the term libertarian has been largely synonymous with anarchism, its meaning has more recently diluted with wider adoption from ideologically disparate groups, including both the New Left and libertarian Marxists, who do not associate themselves with authoritarian socialists or a vanguard party, and extreme cultural liberals, who are primarily concerned with civil liberties. Additionally, some anarchists use libertarian socialist to avoid anarchism\\'s negative connotations and emphasise its connections with socialism. Anarchism is broadly used to describe the anti-authoritarian wing of the so', 'Q: What is the etymological origin of anarchism?\\nA: The etymological origin of anarchism is from the Ancient Greek word \"anarkhia\", meaning \"without a ruler\".']\n",
      "['cialist movement. Anarchism is contrasted to socialist forms which are state-oriented or from above. Scholars of anarchism generally highlight anarchism\\'s socialist credentials and criticise attempts at creating dichotomies between the two. Some scholars describe anarchism as having many influences from liberalism, and being both liberals and socialists but more so, while most scholars reject anarcho-capitalism as a misunderstanding of anarchist principles.\\n\\nWhile opposition to the state is central to anarchist thought, defining anarchism is not an easy task for scholars, as there is a lot of discussion among scholars and anarchists on the matter, and various currents perceive anarchism slightly differently. Major definitional elements include the will for a non-coercive society, the rejection of the state apparatus, the belief that human nature allows humans to exist in or progress toward such a non-coercive society, and a suggestion on how to act to pursue the ideal of anarchy.\\n\\nHistory\\n\\nPre-modern era \\n\\nBefore the establishment of towns and cities, an established authority did not exist. It was after the creation of institutions of authority that anarchistic ideas espoused as a reaction. The most notable precursors to anarchism in the ancient world were in China and Greece. In China, philosophical anarchism (the discussion on the legitimacy of the state) was delineated by Taoist philosophers Zhuang Zhou and Laozi. Alongside Stoicism, Taoism has been said to have had \"significant anticipations\" of anarchism.\\n \\nAnarchic attitudes were also articulated by tragedians and philosophers in Greece. Aeschylus and Sophocles used the myth of Antigone to illustrate the conflict between rules set by the state and personal autonomy. Socrates questioned Athenian authorities constantly and insisted on the right of individual freedom of conscience. Cynics dismissed human law (nomos) and associated authorities while trying to live according to nature (physis). Stoics were supporti', 'Q: What are some major definitional elements of anarchism?\\nA: Major definitional elements of anarchism include the will for a non-coercive society, the rejection of the state apparatus, the belief that human nature allows humans to exist in or progress toward such a non-coercive society, and a suggestion on how to act to pursue the ideal of anarchy.']\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "with open('train_data.json', 'r') as f:\n",
    "    train_data = json.load(f)\n",
    "\n",
    "\n",
    "print(train_data[0])\n",
    "print(train_data[1])\n",
    "print(train_data[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['70698', '69744', '19997', '23319', '75629', '85742', '29056', '23695', '29063', '19854']\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "ids = random.sample(range(10000, 99999), 10)\n",
    "\n",
    "# string\n",
    "ids = [str(i) for i in ids]\n",
    "print(ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from transformers import LlamaTokenizer, AutoModelForCausalLM\n",
    "\n",
    "tokenizer = LlamaTokenizer.from_pretrained(\"syzymon/long_llama_3b\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tokenize the array of ID strings\n",
    "tokenized_ids = tokenizer(ids, return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input_ids': tensor([[    1, 31822, 31888, 31852, 31887, 31877, 31886],\n",
      "        [    1, 31822, 31887, 31877, 31888, 31882, 31882],\n",
      "        [    1, 31822, 31853, 31877, 31877, 31877, 31888],\n",
      "        [    1, 31822, 31855, 31878, 31878, 31853, 31877],\n",
      "        [    1, 31822, 31888, 31880, 31887, 31855, 31877],\n",
      "        [    1, 31822, 31886, 31880, 31888, 31882, 31855],\n",
      "        [    1, 31822, 31855, 31877, 31852, 31880, 31887],\n",
      "        [    1, 31822, 31855, 31878, 31887, 31877, 31880],\n",
      "        [    1, 31822, 31855, 31877, 31852, 31887, 31878],\n",
      "        [    1, 31822, 31853, 31877, 31886, 31880, 31882]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1],\n",
      "        [1, 1, 1, 1, 1, 1, 1],\n",
      "        [1, 1, 1, 1, 1, 1, 1],\n",
      "        [1, 1, 1, 1, 1, 1, 1],\n",
      "        [1, 1, 1, 1, 1, 1, 1],\n",
      "        [1, 1, 1, 1, 1, 1, 1],\n",
      "        [1, 1, 1, 1, 1, 1, 1],\n",
      "        [1, 1, 1, 1, 1, 1, 1],\n",
      "        [1, 1, 1, 1, 1, 1, 1],\n",
      "        [1, 1, 1, 1, 1, 1, 1]])}\n",
      "70698\n"
     ]
    }
   ],
   "source": [
    "print(tokenized_ids)\n",
    "# decode\n",
    "print(tokenizer.decode(tokenized_ids['input_ids'][0][1:]))\n",
    "\n",
    "def generate_batch(data):\n",
    "    ids = random.sample(range(10000, 99999), len(data))\n",
    "    ids = [str(i) for i in ids]\n",
    "    big_string = ''\n",
    "    for i in range(len(data)):\n",
    "        big_string += '\\n[CONTEXT ' + ids[i] + ']\\n' + data[i] + '\\n[/CONTEXT ' + ids[i] + ']\\n'\n",
    "    big_string += '\\nAnswer the following questions, providing highlighted text as evidence for your answer. To cite and highlight, first link the [CONTEXT] block, then add [HIGHLIGHT] tags around the relevant text.\\n'\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}