Poetry

Monday, September 09, 2013

요구사항

1. oracle java : 6에서 최적화됨(2013년 9월 8일 기준)
2. ssh client, sshd 가 각 node 에 설정되어 있고 각 node 들간의 passwordless login 이 가능해야 함
3. dns : hbase.regionserver.dns.interface, 각 hostname 으로 인식, 그러나 없다면,,, 저거 이용,,,
4. 127.0.0.1 : loopback ip
5. ntp 로 time sync 맞춰야 함
6. ulimit 1024 보다 크게

      2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Exception increateBlockOutputStream java.io.EOFException
      2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901

7. hbase 를 실행시키는 user 에 대한 nproc 상향. OutOfMemory 발생할 수 있음
8. hbase version 마다 hadoop 버젼이 다름. 바꿔 말하면 hadoop 버젼마다 지원하는 hbase 버젼이 다름. (S=supported, X=not supported, NT=run, but not tested enough)

able 2.1. Hadoop version support matrix

	HBase-0.92.x	HBase-0.94.x	HBase-0.95
Hadoop-0.20.205	S	X	X
Hadoop-0.22.x	S	X	X
Hadoop-1.0.0-1.0.2^[a]	S	S	X
Hadoop-1.0.3+	S	S	S
Hadoop-1.1.x	NT	S	S
Hadoop-0.23.x	X	S	NT
Hadoop-2.x	X	S	S
^[a]HBase requires hadoop 1.0.3 at a minimum; there is an issue where we cannot find KerberosUtil compiling against earlier versions of Hadoop.

만약 지원하지 않는 hadoop 을 사용하면 hang 이 걸릴 뿐 원인을 알 수 없는 상황이 되어 버릴 수 있음. 주의!!! 또 주의!!!!(standalone 버젼의 경우 내부에 포함된 jar 를 쓰지만 분산된 경우 주의 하지 않으면 안됨. 버젼에 맞게 바꿔줘야함)
9. 각종 버젼에 대한 명시 잘 살펴봐야함. 관련속성도!!
10. 보안접속 : http://hbase.apache.org/book/security.html#hbase.secure.configuration
11. 파일 오픈 개수 제한 dfs.datanode.max.xcievers : 설정 바꿀 경우 반드시 restart(hdfs) : 이 제한이 설정되어 있지 않는 경우 오류(missing block 등)가 발생 할 수 있음

Tuesday, May 31, 2011

An error occured during integration mod_wsgi to django

Django + Apache 이용 설정하면서 기존에 Django 에서 제공되는 테스트 웹서버로 테스트 하던 소스가 제대로 돌아가지 않음을 발견. 에러 로그는 아래와 같다.

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] mod_wsgi (pid=549): Exception occurred processing WSGI script '/Users/heedaepoem/Documents/NoticeTmp/eee.wsgi'.

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] Traceback (most recent call last):

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/Library/Python/2.6/site-packages/django/core/handlers/wsgi.py", line 250, in __call__

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] self.load_middleware()

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/Library/Python/2.6/site-packages/django/core/handlers/base.py", line 45, in load_middleware

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] mod = import_module(mw_module)

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/Library/Python/2.6/site-packages/django/utils/importlib.py", line 35, in import_module

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] __import__(name)

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/Library/Python/2.6/site-packages/django/middleware/transaction.py", line 1, in

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] from django.db import transaction

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/Library/Python/2.6/site-packages/django/db/__init__.py", line 78, in

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] connection = connections[DEFAULT_DB_ALIAS]

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/Library/Python/2.6/site-packages/django/db/utils.py", line 93, in __getitem__

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] backend = load_backend(db['ENGINE'])

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/Library/Python/2.6/site-packages/django/db/utils.py", line 33, in load_backend

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] return import_module('.base', backend_name)

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/Library/Python/2.6/site-packages/django/utils/importlib.py", line 35, in import_module

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] __import__(name)

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/Library/Python/2.6/site-packages/django/db/backends/mysql/base.py", line 11, in

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] import MySQLdb as Database

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/Library/Python/2.6/site-packages/MySQL_python-1.2.3-py2.6-macosx-10.6-universal.egg/MySQLdb/__init__.py", line 19, in

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/Library/Python/2.6/site-packages/MySQL_python-1.2.3-py2.6-macosx-10.6-universal.egg/_mysql.py", line 7, in

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/Library/Python/2.6/site-packages/MySQL_python-1.2.3-py2.6-macosx-10.6-universal.egg/_mysql.py", line 4, in __bootstrap__

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.py", line 835, in resource_filename

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] self, resource_name

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.py", line 1304, in get_resource_filename

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] self._extract_resource(manager, self._eager_to_zip(name))

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.py", line 1326, in _extract_resource

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] self.egg_name, self._parts(zip_path)

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.py", line 915, in get_cache_path

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] self.extraction_error()

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.py", line 881, in extraction_error

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] raise err

[Tue May 31 21:57:32 2011] [error] [client 127.0.0.1] ExtractionError: Can't extract file(s) to egg cache