Script Tool: Process Scene Releases Version 2.0

I wrote in march about how to package a scene release properly. In that post I also published the source of a release creation batch script called ProcRel.bat. I wrote a heavily modified and extended version of that script and present it to you here and today.

NOTE: This Script does not work for SINGLE ZIP release files. It will stop after it created the RAR archive. I have to take a look at it, because it is a minor flaw that does not have to be there, even though it is not the scripts purpose to create releases that are just a single ZIP archive.

Bug Fixes

The script had a bunch of short comings and even a bug. I have not figured out yet what the criteria is for the command line version on WinRAR to create volume numbers of multiple file archives one digit long and when it uses two digits (E.g. FILE.part1.RAR versus FILE.part01.RAR). My old batch assumed 01 instead of the single digit version, but that creates an issue, if RAR decides to use the single digit naming convention instead.

My new version covers this and checks for both versions and then uses the correct one appropriately when it continues by renaming the RAR archives to FILE1.RAR or FILE01.RAR respectively and ZIP Archives afterwards.

Other Minor Enhancements

Also improved was the speed, by utilizing a temporary work directory, which is used to dump all files for each ZIP archive to create the ZIP in one step, without the need to update it with each file (NFO, File_ID.DIZ etc.).

Also included is now the option to include an optional INTO executable.

Major Enhancement 1

But all those changes are minor compared to the really good stuff. The batch does now require a FILE_ID.DIZ template instead of the final File_ID.DIZ file itself. The template should include place holders for the total # of files of the release and a place holder for the current file number. Those place holders will then be replaced by the script automatically with the proper values E.g. [1/10] , [2/10] .... [10/10] etc.

Hell Yeah! It was about time!

Major Enhancement 2

The second major change is that you do not have to mess around with the code of the BATCH file anymore, not do you have to memorize stupid command line parameters to call the script.

All configuration is now done via a separate .INI file. Once you set everything correctly there, you simply execute the Batch either from the command prompt or by double clicking it in Windows Explorer. Simple as  that. :)

I am also so nice this time and let you download the code, including a sample .INI file, file_id.diz (that you can upload it to scene FTP servers or old-school bulletin boards, if you want to), a ReadMe.txt help and documentation text file and a neat little intro for RoySAC.com hehe.

Download the Code Here (ZIP Format) - Backup Download via MediaFire.com

Requirements

You still require the following 3rd party programs, which are not part of this tool (same thing as with the previous version of this tool, nothing new).

• WinRAR, including its Command Line Version RAR.EXE
• WinZip Command Line (NOT included in Standard WinZip Release)
  The File is called WZZIP.EXE and should be located in your Winzip program directory

Usage

Copy the following 2 files into the directory where you would like to create the Release Archive Files:

• ProcRel2.bat
• ProcRel.ini

Open the ProcRel.ini file and modify the parameters as needed.
Once you updated the INI file properly, simply execute the ProcRel2.BAT script
(from the MS DOS Command prompt or simply via double clicking on it in Windows Explorer)

Important Notes!

The batch was not tested for Windows file names. I suggest to use always the short path (DOS 8.3 Name)
The Parameter Names and Section Name in the INI file are "Case-Sensitive" = DON'T CHANGE THEM
Unlike Standard INI Files, do not add spaces before or after the = character which separates the PARAMETER NAME from the VALUE

Correct:
PARAM1=whatever

INCORRECT (might causes BATCH to fail)
PARAM1 = whatever

The RAW INI File Looks like this

[SETTINGS]
ZipExe=
RARExe=
NFOFile=
FILEID=
INTROEXE=
BASEFILENAME=
RAROPT=
RARVOLSIZE=
RAREXTRA=
INPUTDATA=
RARFiles=
IFFILEEXISTS=

The INI Parameters

All Parameter are required, except for INTROEXE and RAREXTRA
If RARFiles has any value except for KEEP (case-sensitive), DEL is assumed as value
If IFFILEEXISTS has any other value than ABORT (case-sensitive), OVERWRITE is assumed as value

ZipExe

Path and File Name to Command Line Version of WinZip
If you installed WinZip to the default directory, the value is probably:
C:\PROGRA~1\WinZip\WZZIP.EXE

RARExe

Path and File Name to the Command Line Version of WinRAR
If you installed WinRAR to the default directory, the value is probably:
C:\PROGRA~1\WinRAR\RAR.EXE

NFOFile

Path and File Name to the NFO file that you want to include with every Release ZIP File
The Path is not required, if you copy the NFO file into the work directory, together with the BAT and INI
Example Values:
RoySAC.nfo
- or-
C:\NFOS\REL1\RoySAC.nfo

FILEID

The path and file name to the template file for the FILE_ID.DIZ
If you copy this file into the work directory, do NOT call it FILE_ID.DIZ or FILE_ID.WRK. It will cause the BATCH to Fail. The Template can contain 2 place holders, which will be replaced by the BATCH at run-time with the actual values.

CURRENTFILE = This string will be replaced with the current archive volume E.g. 1,2,3 ... XX
TOTALFILES = This string in the template will be replaced with the total number of archives that will be created

The Batch copies the Template FILE_ID to the work directory with the Name FILE_ID.WRK and substitutes the string TOTALFILES in it with the actual number of archives created. Then it re-creates the FILE_ID.DIZ for each archive and replaces CURRENTFILE with the number of the current archive.

INTROEXE

Path and File Name of the Intro Executable that you would like to include with every archive, if you have one.

BASEFILENAME

The base file name for your RAR and ZIP archives.  No other Characters than a-z, A-Z,0-9 and the dash (-) and Underscore (_) are allowed. No Spaces, no special Characters. Remember, this is a scene release that you would like to create. The Base File Name will be automatically extended by the archive volume number (1,2,3 ...XX or 01,02 ...0X) and the extension (.RAR and .ZIP)

RAROPT

The main RAR option. The Default is "a", which you should only change, if you want to move the original uncompressed release files into the Archives. In that case, you should use the vale "rn" (without the double quotes) instead. I do not recommend doing this though.

RARVOLSIZE

Specifies the VOLUME size of the archives. You don't want to create just a single gigantic archive, right?
The parameter is specified without the "-" that you would include, if you call RAR from the command line.
It is simply "vXXXXk" or "vXXXXb" where the Xs stand for a number. Depending on the last character being "k" or "b", the meaning of the number is either "(k)ilobytes" or "(b)ytes"

Examples:
v98078k = creates volumes of a bit less than 100 MB to fit on a ZIP-100 disk
v4096k = 4 MB etc.

RAREXTRA

Additional RAR command line options that you might want to add.
The default value is: -m5 -r
The meaning of those two parameters is:

-m5 
sets the compression level. 5 is maximum, you can also set it to 0,1 .. 4 which is faster

-r
includes files in Sub Directories

INPUTDATA

The Path and File Name Filter for the files to be included in the release.
Keep in mind to use the short path for the directory location.
Do not forget the filer, which is typically "*.*"

Example:
C:\DIR\DYPMZA~B\*.*

RARFiles

The default is DEL, meaning that the RAR archives that were created by the BATCH will be deleted, once the Final ZIP archive was created for it. If you would like to keep the RAR archives for any reason, set the value to KEEP (case-sensitive)

IFFILEEXISTS

Use ABORT (case-sensitive), if you would like the batch to abort, if it finds ZIP files with the same BASENAME in the current working directory. This would prevent that you re-create a release, if you had already done so. However, it would also abort, if the ZIP files belong to an incomplete or aborted processing batch. The default is OVERWRITE, which means that the batch deletes all ZIP files with a matching BASENAME in the current
WORK DIRECTORY.

Copyright

This program is FREEWARE! You are free to USE, COPY and even MODIFY it as you see FIT. You are using it at your own risk.

Important Disclaimer & Legal Notice!

The author, of this software accepts no responsibility for damages resulting from the use of this product and makes no warranty or representation, either express or implied, including but not limited to, any implied warranty of merchantability or fitness for a particular purpose.

This software is provided "AS IS", and you, its user, assume all risks when using it.

The Source Code

Download the Code Here (ZIP Format)- Backup Download via MediaFire.com

   1:  @ECHO OFF

   2:  CLS

   3:  REM =================================================

   4:  REM Please Change Name and Path for the Following Variables

   5:  REM =================================================

   6:  SETLOCAL ENABLEDELAYEDEXPANSION

   7:  call:READINI SETTINGS ZipExe

   8:  call:READINI SETTINGS RARExe

   9:  call:READINI SETTINGS NFOFile

  10:  call:READINI SETTINGS FILEID

  11:  call:READINI SETTINGS INTROEXE

  12:  call:READINI SETTINGS BASEFILENAME

  13:  call:READINI SETTINGS RAROPT

  14:  call:READINI SETTINGS RARVOLSIZE

  15:  call:READINI SETTINGS RAREXTRA

  16:  call:READINI SETTINGS INPUTDATA

  17:  call:READINI SETTINGS RARFiles

  18:  call:READINI SETTINGS IFFILEEXISTS

  19:  Echo ---------------------------------------------------------

  20:  Echo ProcRel.INI Parameters

  21:  Echo ---------------------------------------------------------

  22:  Echo ZipExe=%ZipExe%

  23:  Echo RARExe=%RARExe%

  24:  Echo NFOFile=%NFOFile%

  25:  Echo FILEID=%FILEID%

  26:  Echo INTROEXE=%INTROEXE%

  27:  Echo BASEFILENAME=%BASEFILENAME%

  28:  Echo RAROPT=%RAROPT%

  29:  Echo RARVOLSIZE=%RARVOLSIZE%

  30:  Echo RAREXTRA=%RAREXTRA%

  31:  Echo INPUTDATA=%INPUTDATA%

  32:  Echo RARFiles=%RARFiles%

  33:  Echo IFFILEEXISTS=%IFFILEEXISTS%

  34:  Echo ---------------------------------------------------------

  35:  IF NOT EXIST %NFOFile% goto NONFO

  36:  IF NOT EXIST %FILEID% goto NODIZ

  37:  set /a zipf=0

  38:  IF "%IFFILEEXISTS%"=="ABORT" (

  39:     FOR %%Z in (%BASEFILENAME%??.ZIP) DO (

  40:       GOTO ZIPSFOUND

  41:     )

  42:  ) ELSE (

  43:     Del /Q "%BASEFILENAME%??.zip"

  44:     Echo Delete "%BASEFILENAME%??.zip"

  45:  )

  46:  Echo Delete "%BASEFILENAME%.part??.rar" (they should not be there though)

  47:  DEL /Q "%BASEFILENAME%.part??.rar"

  48:  REM =================================================

  49:  Echo Create RAR Archives %1.partXX.RAR for Data Selection %2

  50:  %RARExe% %RAROPT% -%RARVOLSIZE% %RAREXTRA% "%BASEFILENAME%" "%INPUTDATA%"

  51:  REM =================================================

  52:  Echo Rename %BASEFILENAME%.partXX.RAR to %BASEFILENAME%xx.RAR

  53:  set /a pos=0

  54:  For /f %%x in ('dir /on /b "%BASEFILENAME%.part*.rar"') do (

  55:    set /a pos+=1

  56:  )

  57:  if %pos%==0 goto NORAR

  58:  IF NOT EXIST "WORKDIR" MkDir "WORKDIR"

  59:  For /L %%n in (1,1,%pos%) do (

  60:    call :RenRAR %BASEFILENAME% %%n

  61:  )

  62:  Echo.

  63:  Echo =================================================

  64:  Echo %pos% RAR volumes were created!

  65:  Echo =================================================  

  66:  Echo Build List of RAR Archives in Folder...

  67:  dir /B /A:-D *.RAR >!RARFiles.txt

  68:  REM =================================================

  69:  Echo Process RAR Archives...

  70:  Set REP1=TOTALFILES

  71:  IF EXIST "FILE_ID.WRK" DEL /Q "FILE_ID.WRK">NUL

  72:  Call:SandR %FILEID% FILE_ID.WRK %REP1% %pos%

  73:  SET FCNT=0

  74:  Set REP2=CURRENTFILE

  75:  FOR /F "delims=|" %%i IN (!RARFiles.txt) DO (

  76:      Echo Processing %%i

  77:      Set /a FCNT+=1

  78:      IF EXIST "FILE_ID.DIZ" DEL "FILE_ID.DIZ">NUL

  79:      Call:SandR FILE_ID.WRK FILE_ID.DIZ %REP2% !FCNT!

  80:      IF EXIST "%%i.ZIP" del "%%i.ZIP">NUL

  81:      DEL /Q WORKDIR\*.*>NUL

  82:      MOVE "%%i" "WORKDIR\%%i">NUL

  83:      IF EXIST "%NFOFile%" COPY "%NFOFile%" WORKDIR\>NUL

  84:      IF EXIST "FILE_ID.DIZ" COPY "FILE_ID.DIZ" WORKDIR\>NUL

  85:      IF EXIST "%INTROEXE%" COPY "%INTROEXE%" WORKDIR\>NUL

  86:       %ZipExe% -a "%%i.ZIP" "WORKDIR\*.*"

  87:      IF "%RARFiles%" == "KEEP" (MOVE WORKDIR\*.RAR>NUL)

  88:      DEL /Q WORKDIR\*.*>NUL

  89:  )

  90:  REM =================================================

  91:  Echo Build List of TEMP ZIP Files...

  92:  Dir /b *.RAR.ZIP>!ZIPfiles.txt

  93:  REM =================================================

  94:  Echo Rename ZIP Files...

  95:  REM =================================================

  96:  FOR /F "delims=." %%i IN (!ZIPfiles.txt) DO (

  97:      Echo Rename %%i.RAR.ZIP to %%i.ZIP

  98:      IF EXIST "%%i.ZIP" del "%%i.ZIP"

  99:      REN "%%i.RAR.ZIP" "%%i.ZIP"

 100:  )

 101:  REM =================================================

 102:  Echo Deleting TEMP Files...

 103:  IF EXIST !ZIPfiles.txt Del /Q !ZIPfiles.txt

 104:  IF EXIST !RARfiles.txt Del /Q !RARfiles.txt

 105:  IF EXIST "FILE_ID.WRK" DEL /Q "FILE_ID.WRK">NUL

 106:  IF EXIST "FILE_ID.DIZ" DEL /Q "FILE_ID.DIZ">NUL

 107:  RmDir WORKDIR>NUL

 108:  ECHO =================================================

 109:  ECHO Done!

 110:  ECHO =================================================

 111:  Echo.

 112:   Pause

 113:  goto END

 114:   

 115:  REM =============================================

 116:  REM PROCESSING FUNCTIONS

 117:  REM =============================================

 118:  :READINI

 119:  Set key=%~2

 120:  call:strLen key len

 121:  Set /a "len+=1"

 122:  set yfs=

 123:  set ymn=

 124:  for /f "tokens=*" %%i in ( ProcRel.ini ) do (

 125:    set ywv=%%i

 126:    if /i "!ywv!"=="[%1]" (

 127:       set yfs=Y

 128:     ) else (

 129:       set ywv=!ywv:~0,%len%!

 130:       if defined yfs (

 131:          if not defined ymn (

 132:            if /i "!ywv!"=="%2=" ( 

 133:              set ymn=%%i

 134:              set ymn=!ymn:~%len%!

 135:            )

 136:          )

 137:       )

 138:     )

 139:  )

 140:  Set %~2=%ymn%

 141:  GOTO :EOF

 142:  REM =============================================

 143:  :strLen -- returns the length of a str, max 1023

 144:  ::      -- %~1: in - varible name of a string variable

 145:  ::      -- %~2: out- string length

 146:  SETLOCAL

 147:  set str=A!%~1!

 148:  set len=0

 149:  set /a n=1024

 150:  set /a n^>^>=1, len+=n

 151:   if !str:~%len%!. == . set /a len-=n

 152:  set /a n^>^>=1, len+=n

 153:   if !str:~%len%!. == . set /a len-=n

 154:  set /a n^>^>=1, len+=n

 155:   if !str:~%len%!. == . set /a len-=n

 156:  set /a n^>^>=1, len+=n

 157:   if !str:~%len%!. == . set /a len-=n

 158:  set /a n^>^>=1, len+=n

 159:   if !str:~%len%!. == . set /a len-=n

 160:  set /a n^>^>=1, len+=n

 161:   if !str:~%len%!. == . set /a len-=n

 162:  set /a n^>^>=1, len+=n

 163:   if !str:~%len%!. == . set /a len-=n

 164:  set /a n^>^>=1, len+=n

 165:   if !str:~%len%!. == . set /a len-=n

 166:  set /a n^>^>=1, len+=n

 167:   if !str:~%len%!. == . set /a len-=n

 168:  set /a n^>^>=1, len+=n

 169:   if !str:~%len%!. == . set /a len-=n

 170:  ( ENDLOCAL & REM RETURN VALUES

 171:      IF "%~2" NEQ "" SET %~2=%len%

 172:  )

 173:  GOTO:EOF  

 174:  REM =============================================

 175:  :SandR

 176:  REM Search And Replace

 177:  SETLOCAL

 178:  for %%i in (%1) do (

 179:    for /f "tokens=1,* delims=]" %%a in ('"type %%i|find /n /v """') do (

 180:      set "line=%%b"

 181:      if defined line (

 182:          call set "line=echo.%%line:%3=%4%%"

 183:          for /f "delims=" %%X in ('"echo."%%line%%""') do %%~X>>%2

 184:      ) ELSE echo.>>%2

 185:  )

 186:  )

 187:  ENDLOCAL

 188:  goto :eof

 189:  REM =================================================

 190:  :RenRAR

 191:  IF EXIST "%1.part%2.rar" ( 

 192:       ren "%1.part%2.rar" "%1%2.RAR"

 193:       echo ren "%1.part%2.rar" "%1%2.RAR"

 194:  ) else (

 195:     IF EXIST "%1.part0%2.rar" ( 

 196:         ren "%1.part0%2.rar" "%10%2.RAR"

 197:         echo ren "%1.part0%2.rar" "%10%2.RAR"

 198:      )

 199:  )  

 200:  GOTO :EOF

 201:   

 202:  REM =================================================

 203:  REM ERROR MESSAGE FUNCTIONS

 204:  REM =================================================

 205:  :NORAR

 206:  echo.

 207:  Echo Error!

 208:  echo Error! No RAR Archives with the file names

 209:  echo %BASEFILENAME%.partXX.RAR

 210:  echo were Created!

 211:  goto END

 212:  REM =================================================

 213:  :NONFO

 214:  echo.

 215:  Echo Error!

 216:  echo NFO-File %NFOFile% not found!

 217:  goto END

 218:  REM =================================================

 219:  :NODIZ

 220:  echo.

 221:  Echo Error!

 222:  echo File_ID.DIZ Template File NOT Found at %FILEID%

 223:  goto END

 224:  REM =================================================

 225:  :ZIPSFOUND

 226:  echo.

 227:  Echo Error! (ABORT Parameter specified)

 228:  Echo Current directory contains already one or more ZIP files 

 229:  Echo with the name: %BASEFILENAME%??.ZIP 

 230:  goto END

 231:  REM =================================================

 232:  :END

 233:  REM Finished!

 234:  Echo.

 235:  Goto :EOF

Download the Code Here (ZIP Format) - Backup Download via MediaFire.com

Okay, that's it. I hope that you find this little tool/script helpful and practical. As I mentioned indirectly, feel free to spread the ZIP archive around (I would actually appreciate, if you would do so).

Cheers!

Carsten aka Roy/SAC

Data Collection, Sorting, Archiving and Access #1 (Thesis)

babbling my thoughts and theories to nobody and everybody, or in other words, to anybody who might be interested in it :). 

Introduction

I collected myself a vast amount of data and have issues with using them effectively. The lack of ability to find and use specific data that I collected when I need them is a problem that bothers me and where I am constantly working on improving. This problem of mine is of course a problem that pretty much every person has to varying extend, so there are of course numerous options and tools that were created by others already, trying to solve my exact issues.

Considering the fact that virtually everybody has the same problem and that so many "solutions" and tools exist to solve your problem, it surprised me, that virtually of them that I check out failed to solve exactly those problems for ME. This bothered me a lot and got me thinking why this is the case. It didn't make sense at first glance, but after giving it more and more thoughts, I started to realize why most tools that are out there fail to solve the problems for me. 

This is great, but does not help me with actually solving my problem. So I kept thinking about the problem itself, to learn that while the problem appears to be the same for everybody, generally speaking, it is actually not, if you start looking at things a bit more closely. Those differences can make a tool useless for me, but work perfectly fine for somebody else. Many of the differences can also be compensated for, by tools, if the amount of data is relatively small or the available computer resources (hardware, almost every aspect of it) are abundant and can be wasted by getting you what you want, but in a very inefficient manner. 

Pretext

Once the amount of data becomes too large and/or the resources available for the processing too low to get away with wasting tons of them are suddenly not there anymore. At that point the tool(s) that you are using break and become unable to solve your problem. In order to find or create (or inspire somebody else to create) a tool that is truly capable of solving the same problem with its subtle (but eventually significant) differences, for all or at least for most people, we must determine the basic elements and causes of the problems, the goals that need to be accomplished and their reason. 

Then we can think about ways that allow us to solve all those different problems in an efficient way to accomplish the set goals satisfactory.A one-fits-all approach does obviously not work.To accomplish what we want, the tool has to be able to adjust (or be adjusted by the user) to cope with the varying problems and changing facts, such as amount and diversity of data. 

I started with thinking about the root of the problem; the core and heart. I also tried to determine constants, the things that are the same for everybody, because deep inside there is always something that is shared across all the different scenarios that are possible due to the large number of things that can be different and the exponentially increasing number of possible combinations that result from it. 

Thesis

Underlying Context (A Theory)

People collect data throughout their entire life. I use the word "DATA", because information is not brought enough in my opinion. Most DATA collected by human beings are information of some sort. Information are per my own definition data that are classified and structured, viewed from the perspective of each individual human separately. I will use the phrase "Classification and Structure" repeatedly in this document and refer to it as C&S or spell it out, depending on what I feel to be appropriate. 

The C&S system is a collection of indexes and needed to access data quickly. If no C&S system would be in place, the data would have to be processed one by one, until the data that are currently needed are found. Data are like a pile of small boxes, where each box contains a junk. "C", the Classification,  would be labels put on the box, with keywords that describe the content. "S", the structure, would be stacks of boxes that share a common keyword, grouped in columns by even broader keywords, then rooms, floors, buildings, facilities etc. 

The same data might be classified and structured for one person, but not for another. Classification and Structure is stored separately from the Data, because every human develops his own framework and classification structure, designed for his current needs, priorities and understandings of the world. This framework is constantly changing and also expanding throughout the entire life time of each person. It is not and will never be static and fixed.

Of course, if classifications and structure can be found for the data that a person collects, it could be the case that it is identical or partially compatible with the persons own classification system and structure. In some cases it can be adapted (replicated) as is, but in many cases is it necessary to modify the C&S found with the data before it is integrated into the existing C&S. Different terms might be used for the same thing or the structure needs to be diversified or simplified. (See the box with my example for clarifications and illustration) 

 

Example: Differences of the C&S used for ONE set of Data by Different People A set of geological data about the moons of the planet Saturn. The data were compiled by a physicist/astronomer who is working for an university. The astronomer is a member of NASA's Cassini mission team and his full job is to work with the geological data collected by the probe about the moons. Since the physicist is an expert, who was working for years already on this specific subject and thus makes up a huge portion of that persons life, the data related to this subject are structured very deeply and many scientific classifications are used where most ordinary people never heard of. As most NASA mission data, this geological data are published on the NASA web site and made accessible to the public. A person interested in astronomy comes across those data and decides to "store" some parts or all of it in his own archive. The classifications used by the scientist would most likely too specific and replaced by more generic ones instead. The structure would probably be flattened and simplified as well. While the physicist distinguishes moons by very specific geological properties, like by type of material the moons are made of, or properties like "geologically active" or not, with atmosphere or without etc., the person does not, classifying all of them simply as "moon of planet Saturn".

A Bio-Technical Marvel

The data with the corresponding C&S is stored in the human brain (memory). The human brain is a marvelous archiving system with a huge storage capacity. But the capacity is not limitless. It has two different sets where it can store the data, which work differently. There is the set that contains recently collected data and data that were recently used, but collected a long time ago already.The access to those data is very fast and ready to use  in an instant, if needed.The second set, with the largest capacity, holds data that were collected a long time ago and not used for a while either. 

The long term memory, the humans data archive and vault. Most data stored here take some time to access, many even require that the right trigger (Tag, Specific Classification) is used to even find the data itself. Usually data that were not used for while that are stored in the short and mid-term memory, are "moved' to the archive of the long-term memory and then moved to the deep areas of the archive, the one that requires the use of the right triggers to access them again. The brain automatically determines the relevance of each data and drops the ones it believes to be not to be relevant anymore. 

Every Has Flaws

This marvel of biological engineering has some serious flaws though. Classification of data happens based on the current C&S system used by the person. Data are also altered (mostly shrunk) over the course of time and their move between or within the 2 memory sets.It is virtually impossible to determine now if and when a set of data might be required in the future again. Vital data might get dropped or shrunk and thus unavailable or useless, once we need them again or the classification system changed so much, that it became hard to impossible to find and deliberately use the right triggers to access data that archived deep in the long term memory

Humans are perfectly aware of this problem and used various methods to compensate for those limitations and shortcomings. Things, physical objects, that can act as a trigger to access data in the distant future can be kept and stored somewhere and used when needed. Those object might even contain some or all of the data itself. When writing was invented for example, documents could be created that contain all the data, so the object would not just be a trigger, it would hold the entire data itself and allow the complete reconstruction, if the data that were kept in a persons memory were erased, shrunk or altered. Storing just the data is not enough, if the amount of data comes very large. Accessing a specific one when needed is not easy to virtually impossible.

The Catch 

The C&S must be stored outside the brain, together with the data in order to be able to access data again in a reasonable amount of time. The C&S attached to the data can also be stored like a written document outside the human brain of course. 

There is a problem though. The C&S system of a person changes over time. It is possible that it changed so much that the C&S used in the past and stored with the data became virtually incompatible to the current C&S system in use.It would make the data that were stored as inaccessible as if no C&S would have been stored with them in the first place.

The C&S of data in the short, mid-term and to a limited extend also the long-term memory are updated by the human brain automatically, if a change in the C&S system of a person occurs. This update must also be done, manually, with the C&S of the data that were stored "offline" as backup. Updating it every time there is a change would be insufficient and consume too much time. It should be done though, before the number and type of chances become too much to be applied without loss to the "offline" archive. 

A Look at Human History

Humans created archives with data that are relevant and/or vital for the society as whole. Those archives are typically administered by people who's job it is to keep the C&S system used for the archive up to date and most importantly help other people who would like to use data from that archive to find them, by assisting with the matching of that persons C&S systems to the one used by the archive, exact matching preferably, but at least partially only a smaller set of data might have to be processed one after another to find the one that matches.

A New Age

Now we entered the digital age and the access, replication and storage of vast amounts of data becomes easier, while the amount of data that is created or made accessible grows rapidly at the same time. The limitations of physical archives in terms of physical storage space to keep it and the time and cost associated with the creation of the documents are multiple times less when done electronically. Thanks to the rapid advancements in technologies, this reduction of limitations continues in an exponential rate. 

People can now create vast archives that contain data of interest to them on a scale unheard of in human history. The climax would be reached, if ALL DATA in existence would be accessible to ANYBODY in an instant at ANY TIME, which is of course only hypothetical, because this will NEVER happen. As long as words such as privacy, intellectual property and secret have any meaning in human society, ALL DATA will never become accessible by ANYBODY. Some data will always be restricted to one individual person or restricted and limited group of people. This means that there cannot be just a single data archive for everything without the need to repeated replication. 

Back to Reality

The reality is that even today, some data that are accessible today, might not so in the future for different reasons. So if a person accesses data today and deems it to be relevant and important to be able to access in the future again, this person will replicate the data and save them somewhere, where he has control of it (add, edit and delete) and access to (read) whenever needed.

Even if people believe that access will be available in the future and replication of them not required for a large amount of data, the problem of C&S of those data remains. The creation of a general C&S system for this huge and centralized digital data archive cannot substitute the C&S system in place by an individual. It would be way too complex and detailed and thus be useless as an index for fast access.

Conclusion

The C&S system of the person only has the size and dimensions that an individual needs today. Refinements are made in areas where more data are collected and used, such as when a person becomes a specialist in something. Structures are simplified if the data become less important and/or less frequently used, for example in the case where a person changes his career or if an once important hobby becomes less interesting and fades away, maybe because something else is pursuit  now instead.

Does this make sense to anybody but myself? Are my assumptions and speculations making sense or am I missing the mark entirely? Voice your thoughts and opinions via the comments section below, openly or anonymously.

Cheers!

Carsten aka Roy/SAC

The Fall of the Berlin Wall - Personal Account of Events

In response to my videos about the Berlin Wall (like this one on Vimeo.com) or when people who I meet in person learn that I grew up in the former East-Berlin (the communist part of the city), they often ask me about my own personal experiences and feelings that I had.

Specifically when it comes to the time when I was 15 years old and from when the Wall fell on November 9, 1989 until the reunion of the two German segments on October 3, 1990; a time that was emotionally and turbulent with new events unfolding rapidly, like a land slide or avalanche.

Notes and Clarifications

I posted pretty much the same stuff as comments to a previous blog post of mine, but thought to myself that it actually warrants the creation of an individual post just dedicated to this.

I also posted already the publicly known historic facts of the events. This one is personal and I tried to recall what I thought at that time and what happened from my own perspective as good and accurate as I can. It has been almost 20 years since they actually occurred, so some of the details of my memories could be flawed and incorrect (that is not because I intended that, but because it is a “human flaw” when it comes to “storing” and “cataloging” personal long term memories.

Flashback Start

I personally was surprised by the events that unfolded. I did not realize what happened in the night of November 9, 1989 and went to school as always on November 10. Half the school wasn't there; even some teachers were not there. Nobody knew what was going on.

Around noon some kids showed up and asked questions like

"Hey, have you been to the 'Kurfürsten Damm' too?"

(That was the “Main Street” and kind of new center of West Berlin). I thought that they joked and laughed, but then other kids came talking about the same stuff. I realized that they couldn't have possibly made the story up together and then come to school in groups, 15-60 minutes apart from each other.

Somebody mentioned that there is a visa required, so I and 2 other friends went to the local police station and asked about it. They acknowledged it and we got out IDs stamped (You got your ID with the age of 14 in East Germany).

We went to the border crossing “Bornholmer Straße”, but the border was shut. That did not surprise me, because I could not imagine that they just open the border over night and that it must have been a fluke the night before.

Well, we got there around 3 PM or so and waited like the rest to see what is going to happen. The visa in my ID stated "valid from November 11, 1989", the next day, but most other folks at the border didn't even have a visa at all and still waited there and shouted to the border guards.

I don't know how long we waited, but the number of people started to grow rapidly at one point. I guess a lot of people got off work and flocked to the border, just as we did, only after school (which was off around 2 PM).

They let some people pass, one by one, but that wasn't enough. The increasing number of people, which all pushed towards the border created problems, because the first rows were smashed together harder and harder and it was only a matter of time, before anybody would have been injured or worse.

So they opened the gate again and we flushed through the border in this bulk of people. Before we knew it, we were on West Berliner side of the border. It felt weird and for me like an entirely different city. I did not know anything about West Berlin, there were no maps. I only had one from 1960 or so. I thought about finding my way to some family members who lived in West Berlin, but then decided to stick with my friends.

Welcome Money for East Germans

We heard about the 100 DM free welcome money for East Germans who visited West Germany or West Berlin and followed some folks to a bank that was nearby. The banks were of course completely overwhelmed by the flood of people that suddenly appeared to pick up their "present".

The 100 DM were originally meant for the few East Germans that were allowed to travel to West Germany, like elderly people. They had no West German money, so they could get once per year this free money, what they usually spent on little things for their kids and grand kids (“Cold War” strategy, obviously hehe.)

The banks didn't make it a formality that day and gave out the 100 DM without any paperwork to be filled out. They stamped my ID with a "Date Stamp" that could have been anything and that was it.

I don't know exactly what I bought that day with my free welcome money. I think it was a pack of TIC-TAC, a Coke, a Comic and maybe some chocolate bar. Junk actually, stuff we had in East Germany too, but different (West German TV advertisement had its influence on East-Germans. I don't know if anybody took the time to study this kind of stuff. Would have been a gold mine for advertisers and marketers to find out what people thought about products after years of exposure by TV advertisement, without much additional exposure to anything else and with rare to no opportunity to test, taste, touch the actual product itself. Interesting thought :).)

Opinions about the Reunion Process

I am glad that the wall is gone (and NO, I don't have nor want a piece of it, like most Berliners). How the "Anschluß" or join/merge of East Germany into West Germany was handled was bad. It was driven by emotions and not by realistic thoughts. That's the wrong thing to do when it comes to things of such scale and importance that have lasting effects well after the time, when emotions cooled down again. Everybody knows that, but many folks forgot about it back then. The people who knew the best were fired, because their suggestions did not fit the emotional climate. This mistake ended up to be a very costly one for Germany as a whole.

I remember vividly how my dad, who was interested in, as well as knowledgeable about economics (East and West-German), was talking about all the bad stuff that is going to happen, if a reunion is being done so quickly and in the way people wanted to do it. It sounded grim, to some degree even horrific and mom mad him shut up when he started to talk about it in public to other people, because their reaction to this was typically first disbelieve, followed by hostility and accusations that my dad is just an old-school regime-loyal Stalinist, who would love to see the border shut again and the Wall raised by a few extra meters instead.

This accusation was typical for anybody who voiced concerns about how things are planned and supposed to work out right. Most of those people were everything else but regime-loyal and/or Stalinists. They just made the mistake to think though some of the cold facts with leaving emotions out of this process. Yes, this is of course the generally well known and suggested approach to making decisions that are complex and will have long term consequences. Rule #1: Do not make any such decisions while you are heavily emotional biased and unable to form clear thoughts about the subject matter.

Emotions are usually short lived, but long term decisions made during this short live span are staying with you. (A good example is getting a real tattoo with the name of your “eternal love” who you only know for a couple days, weeks or months, which ends up being not so much eternal as you thought at the time. Eternal is a damn long time, even if it is only the short live span of a human being. The love fades away maybe within weeks or months but the tattoo fades, which also fades, does it at a so slow rate that it won’t disappear completely during your natural life time. Undoing what you did in a hurry while emotionally loaded is hard to impossible. Even plastic surgery usually cannot un-do it to the status where it was originally in most cases, removing the visible Name, only to replace it with a different looking reminder of your stupidity.

Conclusion

I was probably affected by this the least, because I was young and also not stupid. For me a whole world of possibilities opened up over night. Especially my dream to have my own computer and to do something with computers professionally was fulfilling itself faster and better than I could have imagined it just a few years before.

I hope you enjoyed this personal account of those historically significant world events. Let me know via the comments section of this blog post further down below. Cheers!

Carsten aka Roy/SAC

More Berlin Wall Articles of Mine

• The Berlin Wall and the Walls between Us (August 6, 2008)
• The Berlin Wall History - Lessons Learned ... Again (September 4, 2008)
• Moments in History - The Fall of the Berlin Wall (December 6, 2008)
• The Berlin Wall - August 13, 1961 to November 9, 1989 (July 29, 2009)

Videos of Mine About the Subject

My "Berlin Wall - Lessons Learned" Videos

• Version 1 (08:53 min)
• Version 2 (09:50 min)
• Version 3 (13:01 min)
• Version 4 (30:13 min) or at Archive.org

and my (pretty popular) Video

• Moments in History - The Fall of the Berlin Wall (5:56 min) or at Archive.org

Also May Be of Interest

• My Berlin Wall Videos Downloads at Mediafire.com
• Additional Video Material Downloads at Mediafire.com
• Flickr Photosets
  • Berlin Wall Photographs
  • East Berlin Photographs
  • East German Money – Mark and Pfennige