16_RNN_Attention_q.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = [\"April 22,2019\", \"March 10,2019\", \"June 09,2019\", \"March 11,2021\", \"March 12,2019\"]\n",
    "y = [\"2019-04-22\", \"2019-03-10\", \"2019-06-09\", \"2021-03-11\", \"2019-03-12\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Embedding(input_dim=len(x) + 1,\n",
    "                           output_dim=embedding_size,\n",
    "                           input_shape=[None]),\n",
    "    keras.layers.LSTM(128)\n",
    "])\n",
    "\n",
    "decoder = keras.models.Sequential([\n",
    "    keras.layers.LSTM(128, return_sequences=True),\n",
    "    keras.layers.Dense(len())\n",
    "])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First version: a very basic seq2seq model\n",
    "\n",
    "Second version: feeding the shifted targets to the decoder (teacher forcing)\n",
    "\n",
    "Third version: using TF-Addons's seq2seq implementation\n",
    "\n",
    "Fourth version: using TF-Addons's seq2seq implementation with a scheduled sampler\n",
    "\n",
    "Fifth version: using TFA seq2seq, the Keras subclassing API and attention mechanisms"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 첫 번째 버전: 매우 기본적인 seq2seq 모델\n",
    "이 모델은 기본적인 seq2seq 모델입니다. 인코더는 입력 시퀀스를 입력 받고 마지막 시간 단계에서 컨텍스트 벡터를 출력하며, 디코더는 이전 시간 단계의 출력을 현재 시간 단계의 입력으로 사용하여 시퀀스를 생성합니다.\n",
    "\n",
    "인코더는 입력 문자열을 임베딩하고, LSTM 레이어를 사용하여 입력 시퀀스를 인코딩합니다.\n",
    "전체 모델은 인코더, RepeatVector 레이어(출력 시퀀스의 길이와 맞추기 위해 입력 시퀀스를 반복), 디코더로 구성됩니다.\n",
    "\n",
    "> 두 번째 버전: 디코더에 시프트된 타겟을 입력하는 teacher forcing\n",
    "이 모델은 두 번째 버전입니다. 이 모델은 첫 번째 버전과 거의 동일하지만, 디코더는 이전 시간 단계에서의 출력 대신 시프트된 타겟 시퀀스를 현재 시간 단계의 입력으로 사용합니다. 이 방법을 \"teacher forcing\" 이라고 합니다. 이렇게 하면 디코더가 예측한 출력을 다음 시간 단계에서의 입력으로 사용하지 않고 실제 타겟 값을 사용하므로 학습이 더 빠르고 안정적으로 이루어집니다.\n",
    "\n",
    "> 세 번째 버전: TF-Addons의 seq2seq 구현 사용\n",
    "이 모델은 TF-Addons의 seq2seq 구현을 사용합니다. 이 구현은 첫 번째와 두 번째 모델과 비슷하지만, 좀 더 일반화된 구현을 제공합니다. 또한 이 구현에서는 입력과 출력 시퀀스의 길이를 일치시키기 위해 패딩이 사용됩니다.\n",
    "\n",
    "> 네 번째 버전: 스케줄된 샘플러를 사용한 TF-Addons의 seq2seq 구현\n",
    "이 모델은 TF-Addons의 seq2seq 구현을 사용합니다. 이 구현에서는 스케줄된 샘플링을 사용하여 디코더의 입력을 생성합니다. 이 방법을 사용하면 디코더가 일정 비율로 이전 시간 단계의 출력과 타겟 값을 사용하므로, 더 정확한 예측을 할 수 있습니다.\n",
    "\n",
    "> 다섯 번째 버전: TFA seq2seq, Keras 서브클래싱 API 및 어텐션 메커니즘 사용\n",
    "이 모델은 TFA seq2seq, Keras 서브클래싱 API 및 어텐션 메커니즘을 사용합니다. 이 모델은 이전 버전에서 사용된 모든 기술을 결합하며, 어텐션 메커니즘을 사용하여 디코더가 인코더의 출력 중에서 어떤 부분에 더 집중해야있는지 결정합니다. 이 모델은 Keras의 서브클래싱 API를 사용하여 모델을 정의하며, TFA seq2seq 구현을 사용하여 학습을 수행합니다. 이 모델은 이전 버전에 비해 더 높은 성능을 제공할 수 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR:root:Internal Python error in the inspect module.\n",
      "Below is the traceback from this internal error.\n",
      "\n",
      "ERROR:root:Internal Python error in the inspect module.\n",
      "Below is the traceback from this internal error.\n",
      "\n",
      "ERROR:root:Internal Python error in the inspect module.\n",
      "Below is the traceback from this internal error.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 3457, in run_code\n",
      "    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "  File \"/var/folders/0s/knzc6bzs1l9dgw6x4k24t9740000gn/T/ipykernel_64292/1662734598.py\", line 20, in <module>\n",
      "    def date_str_to_ids(date_str, chars=INPUT_CHARS):\n",
      "NameError: name 'INPUT_CHARS' is not defined\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 2077, in showtraceback\n",
      "    stb = value._render_traceback_()\n",
      "AttributeError: 'NameError' object has no attribute '_render_traceback_'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1101, in get_records\n",
      "    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 248, in wrapped\n",
      "    return f(*args, **kwargs)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 281, in _fixed_getinnerframes\n",
      "    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/inspect.py\", line 1502, in getinnerframes\n",
      "    frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/inspect.py\", line 1460, in getframeinfo\n",
      "    filename = getsourcefile(frame) or getfile(frame)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/inspect.py\", line 696, in getsourcefile\n",
      "    if getattr(getmodule(object, filename), '__loader__', None) is not None:\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/inspect.py\", line 733, in getmodule\n",
      "    if ismodule(module) and hasattr(module, '__file__'):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/tensorflow/__init__.py\", line 50, in __getattr__\n",
      "    module = self._load()\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/tensorflow/__init__.py\", line 44, in _load\n",
      "    module = _importlib.import_module(self.__name__)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/importlib/__init__.py\", line 127, in import_module\n",
      "    return _bootstrap._gcd_import(name[level:], package, level)\n",
      "  File \"<frozen importlib._bootstrap>\", line 1006, in _gcd_import\n",
      "  File \"<frozen importlib._bootstrap>\", line 983, in _find_and_load\n",
      "  File \"<frozen importlib._bootstrap>\", line 965, in _find_and_load_unlocked\n",
      "ModuleNotFoundError: No module named 'tensorflow_core.estimator'\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 3457, in run_code\n",
      "    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "  File \"/var/folders/0s/knzc6bzs1l9dgw6x4k24t9740000gn/T/ipykernel_64292/1662734598.py\", line 20, in <module>\n",
      "    def date_str_to_ids(date_str, chars=INPUT_CHARS):\n",
      "NameError: name 'INPUT_CHARS' is not defined\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 2077, in showtraceback\n",
      "    stb = value._render_traceback_()\n",
      "AttributeError: 'NameError' object has no attribute '_render_traceback_'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 3377, in run_ast_nodes\n",
      "    if (await self.run_code(code, result,  async_=asy)):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 3474, in run_code\n",
      "    self.showtraceback(running_compiled_code=True)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 2080, in showtraceback\n",
      "    value, tb, tb_offset=tb_offset)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1368, in structured_traceback\n",
      "    self, etype, value, tb, tb_offset, number_of_lines_of_context)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1268, in structured_traceback\n",
      "    self, etype, value, tb, tb_offset, number_of_lines_of_context\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1125, in structured_traceback\n",
      "    tb_offset)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1082, in format_exception_as_a_whole\n",
      "    last_unique, recursion_repeat = find_recursion(orig_etype, evalue, records)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 382, in find_recursion\n",
      "    return len(records), 0\n",
      "TypeError: object of type 'NoneType' has no len()\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 2077, in showtraceback\n",
      "    stb = value._render_traceback_()\n",
      "AttributeError: 'TypeError' object has no attribute '_render_traceback_'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1101, in get_records\n",
      "    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 248, in wrapped\n",
      "    return f(*args, **kwargs)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 281, in _fixed_getinnerframes\n",
      "    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/inspect.py\", line 1502, in getinnerframes\n",
      "    frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/inspect.py\", line 1460, in getframeinfo\n",
      "    filename = getsourcefile(frame) or getfile(frame)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/inspect.py\", line 696, in getsourcefile\n",
      "    if getattr(getmodule(object, filename), '__loader__', None) is not None:\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/inspect.py\", line 733, in getmodule\n",
      "    if ismodule(module) and hasattr(module, '__file__'):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/tensorflow/__init__.py\", line 50, in __getattr__\n",
      "    module = self._load()\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/tensorflow/__init__.py\", line 44, in _load\n",
      "    module = _importlib.import_module(self.__name__)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/importlib/__init__.py\", line 127, in import_module\n",
      "    return _bootstrap._gcd_import(name[level:], package, level)\n",
      "  File \"<frozen importlib._bootstrap>\", line 1006, in _gcd_import\n",
      "  File \"<frozen importlib._bootstrap>\", line 983, in _find_and_load\n",
      "  File \"<frozen importlib._bootstrap>\", line 965, in _find_and_load_unlocked\n",
      "ModuleNotFoundError: No module named 'tensorflow_core.estimator'\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 3457, in run_code\n",
      "    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "  File \"/var/folders/0s/knzc6bzs1l9dgw6x4k24t9740000gn/T/ipykernel_64292/1662734598.py\", line 20, in <module>\n",
      "    def date_str_to_ids(date_str, chars=INPUT_CHARS):\n",
      "NameError: name 'INPUT_CHARS' is not defined\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 2077, in showtraceback\n",
      "    stb = value._render_traceback_()\n",
      "AttributeError: 'NameError' object has no attribute '_render_traceback_'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 3377, in run_ast_nodes\n",
      "    if (await self.run_code(code, result,  async_=asy)):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 3474, in run_code\n",
      "    self.showtraceback(running_compiled_code=True)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 2080, in showtraceback\n",
      "    value, tb, tb_offset=tb_offset)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1368, in structured_traceback\n",
      "    self, etype, value, tb, tb_offset, number_of_lines_of_context)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1268, in structured_traceback\n",
      "    self, etype, value, tb, tb_offset, number_of_lines_of_context\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1125, in structured_traceback\n",
      "    tb_offset)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1082, in format_exception_as_a_whole\n",
      "    last_unique, recursion_repeat = find_recursion(orig_etype, evalue, records)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 382, in find_recursion\n",
      "    return len(records), 0\n",
      "TypeError: object of type 'NoneType' has no len()\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 2077, in showtraceback\n",
      "    stb = value._render_traceback_()\n",
      "AttributeError: 'TypeError' object has no attribute '_render_traceback_'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 2960, in _run_cell\n",
      "    return runner(coro)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/async_helpers.py\", line 78, in _pseudo_sync_runner\n",
      "    coro.send(None)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 3186, in run_cell_async\n",
      "    interactivity=interactivity, compiler=compiler, result=result)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 3396, in run_ast_nodes\n",
      "    self.showtraceback()\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 2080, in showtraceback\n",
      "    value, tb, tb_offset=tb_offset)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1368, in structured_traceback\n",
      "    self, etype, value, tb, tb_offset, number_of_lines_of_context)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1268, in structured_traceback\n",
      "    self, etype, value, tb, tb_offset, number_of_lines_of_context\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1143, in structured_traceback\n",
      "    chained_exceptions_tb_offset)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1082, in format_exception_as_a_whole\n",
      "    last_unique, recursion_repeat = find_recursion(orig_etype, evalue, records)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 382, in find_recursion\n",
      "    return len(records), 0\n",
      "TypeError: object of type 'NoneType' has no len()\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/interactiveshell.py\", line 2077, in showtraceback\n",
      "    stb = value._render_traceback_()\n",
      "AttributeError: 'TypeError' object has no attribute '_render_traceback_'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 1101, in get_records\n",
      "    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 248, in wrapped\n",
      "    return f(*args, **kwargs)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/IPython/core/ultratb.py\", line 281, in _fixed_getinnerframes\n",
      "    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/inspect.py\", line 1502, in getinnerframes\n",
      "    frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/inspect.py\", line 1460, in getframeinfo\n",
      "    filename = getsourcefile(frame) or getfile(frame)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/inspect.py\", line 696, in getsourcefile\n",
      "    if getattr(getmodule(object, filename), '__loader__', None) is not None:\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/inspect.py\", line 733, in getmodule\n",
      "    if ismodule(module) and hasattr(module, '__file__'):\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/tensorflow/__init__.py\", line 50, in __getattr__\n",
      "    module = self._load()\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/tensorflow/__init__.py\", line 44, in _load\n",
      "    module = _importlib.import_module(self.__name__)\n",
      "  File \"/Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/importlib/__init__.py\", line 127, in import_module\n",
      "    return _bootstrap._gcd_import(name[level:], package, level)\n",
      "  File \"<frozen importlib._bootstrap>\", line 1006, in _gcd_import\n",
      "  File \"<frozen importlib._bootstrap>\", line 983, in _find_and_load\n",
      "  File \"<frozen importlib._bootstrap>\", line 965, in _find_and_load_unlocked\n",
      "ModuleNotFoundError: No module named 'tensorflow_core.estimator'\n"
     ]
    }
   ],
   "source": [
    "\n",
    "from datetime import date\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "# cannot use strftime()'s %B format since it depends on the locale\n",
    "MONTHS = [\"January\", \"February\", \"March\", \"April\", \"May\", \"June\",\n",
    "          \"July\", \"August\", \"September\", \"October\", \"November\", \"December\"]\n",
    "\n",
    "def random_dates(n_dates):\n",
    "    min_date = date(1000, 1, 1).toordinal()\n",
    "    max_date = date(9999, 12, 31).toordinal()\n",
    "\n",
    "    ordinals = np.random.randint(max_date - min_date, size=n_dates) + min_date\n",
    "    dates = [date.fromordinal(ordinal) for ordinal in ordinals]\n",
    "\n",
    "    x = [MONTHS[dt.month - 1] + \" \" + dt.strftime(\"%d, %Y\") for dt in dates]\n",
    "    y = [dt.isoformat() for dt in dates]\n",
    "    return x, y\n",
    "\n",
    "def date_str_to_ids(date_str, chars=INPUT_CHARS):\n",
    "    return [chars.index(c) for c in date_str]\n",
    "\n",
    "np.random.seed(42)\n",
    "\n",
    "n_dates = 3\n",
    "x_example, y_example = random_dates(n_dates)\n",
    "print(\"{:25s}{:25s}\".format(\"Input\", \"Target\"))\n",
    "print(\"-\" * 50)\n",
    "for idx in range(n_dates):\n",
    "    print(\"{:25s}{:25s}\".format(x_example[idx], y_example[idx]))\n",
    "    \n",
    "INPUT_CHARS = \"\".join(sorted(set(\"\".join(MONTHS) + \"0123456789, \")))\n",
    "OUTPUT_CHARS = \"0123456789-\"\n",
    "\n",
    "def prepare_date_strs(date_strs, chars=INPUT_CHARS):\n",
    "    X_ids = [date_str_to_ids(dt, chars) for dt in date_strs]\n",
    "    X = tf.ragged.constant(X_ids, ragged_rank=1)\n",
    "    return (X + 1).to_tensor() # using 0 as the padding token ID\n",
    "\n",
    "def create_dataset(n_dates):\n",
    "    x, y = random_dates(n_dates)\n",
    "    return prepare_date_strs(x, INPUT_CHARS), prepare_date_strs(y, OUTPUT_CHARS)\n",
    "\n",
    "np.random.seed(42)\n",
    "\n",
    "X_train, Y_train = create_dataset(10000)\n",
    "X_valid, Y_valid = create_dataset(2000)\n",
    "X_test, Y_test = create_dataset(2000)\n",
    "\n",
    "sos_id = len(OUTPUT_CHARS) + 1\n",
    "\n",
    "def shifted_output_sequences(Y):\n",
    "    sos_tokens = tf.fill(dims=(len(Y), 1), value=sos_id)\n",
    "    return tf.concat([sos_tokens, Y[:, :-1]], axis=1)\n",
    "\n",
    "X_train_decoder = shifted_output_sequences(Y_train)\n",
    "X_valid_decoder = shifted_output_sequences(Y_valid)\n",
    "X_test_decoder = shifted_output_sequences(Y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.15.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf; print(tf.__version__)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}